{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "markdown-1",
   "metadata": {},
   "source": [
    "# 1. What was known, and when\n",
    "\n",
    "A forecast is a guess about tomorrow made with what we know today. That sounds\n",
    "too obvious to say. Yet it is the easiest rule to break by accident: a model\n",
    "that peeks at tomorrow's prices while \"forecasting\" them looks brilliant on\n",
    "paper and is useless in practice.\n",
    "\n",
    "So the platform keeps two times for every value:\n",
    "\n",
    "- when it is **valid**: the hour the price is for;\n",
    "- when it became **known**: the moment it was published.\n",
    "\n",
    "Every read asks one question: *what did we know at this moment?* That moment is\n",
    "the **cut-off**, called `as_of`.\n",
    "\n",
    "This notebook shows the idea on real prices. Run each cell with Shift+Enter.\n",
    "Before you run one, guess what it will show; that is how the idea sticks."
   ]
  },
  {
   "cell_type": "code",
   "id": "code-2",
   "metadata": {},
   "source": [
    "import pandas as pd\n",
    "\n",
    "from bff import lab"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "markdown-3",
   "metadata": {},
   "source": [
    "## The prices of one day\n",
    "\n",
    "Germany's day-ahead auction closes at noon. At about 12:45 it publishes a price\n",
    "for each hour of the next day. Here are the prices for Thursday, 2 March 2017,\n",
    "as we know them today.\n",
    "\n",
    "The platform speaks UTC. Midnight in Berlin was 23:00 UTC on 1 March, so the\n",
    "day runs from there to 23:00 UTC on 2 March."
   ]
  },
  {
   "cell_type": "code",
   "id": "code-4",
   "metadata": {},
   "source": [
    "DAY_START = \"2017-03-01T23:00:00Z\"\n",
    "DAY_END = \"2017-03-02T23:00:00Z\"\n",
    "\n",
    "prices = lab.series(\"epf.price\", DAY_START, DAY_END)\n",
    "prices.head()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "markdown-5",
   "metadata": {},
   "source": [
    "Each row is one hour's price, in euros per megawatt-hour (EUR/MWh). The index\n",
    "is the hour, in UTC. To read it on the market's clock, convert it:"
   ]
  },
  {
   "cell_type": "code",
   "id": "code-6",
   "metadata": {},
   "source": [
    "prices.tz_convert(\"Europe/Berlin\").head(3)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "markdown-7",
   "metadata": {},
   "source": [
    "## The same day, as known the morning before\n",
    "\n",
    "Now go back in time to 09:00 in Berlin on 1 March, before the auction. How many\n",
    "prices for 2 March did anyone know then? Guess, then run the cell. (09:00 in\n",
    "Berlin is 08:00 UTC in winter.)"
   ]
  },
  {
   "cell_type": "code",
   "id": "code-8",
   "metadata": {},
   "source": [
    "morning = lab.series(\"epf.price\", DAY_START, DAY_END, as_of=\"2017-03-01T08:00:00Z\")\n",
    "len(morning)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "markdown-9",
   "metadata": {},
   "source": [
    "None. The auction had not run, so those prices did not exist yet. The platform\n",
    "does not guess or fill them in; it answers with what was known.\n",
    "\n",
    "Ask again at 13:00 in Berlin (12:00 UTC), after the auction:"
   ]
  },
  {
   "cell_type": "code",
   "id": "code-10",
   "metadata": {},
   "source": [
    "afternoon = lab.series(\"epf.price\", DAY_START, DAY_END, as_of=\"2017-03-01T12:00:00Z\")\n",
    "len(afternoon)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "markdown-11",
   "metadata": {},
   "source": [
    "All 24. The cut-off changes what you can see. It never changes what the values\n",
    "were.\n",
    "\n",
    "## When each value became known\n",
    "\n",
    "`lab.read` returns the same values with a `known_at` column: the moment each\n",
    "one was published."
   ]
  },
  {
   "cell_type": "code",
   "id": "code-12",
   "metadata": {},
   "source": [
    "rows = lab.read([\"epf.price\", \"epf.load_forecast\"], DAY_START, DAY_END)\n",
    "rows.groupby(\"name\")[\"known_at\"].unique()"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "markdown-13",
   "metadata": {},
   "source": [
    "The prices became known at 11:45 UTC, which is 12:45 in Berlin. The forecast of\n",
    "the next day's load, `epf.load_forecast`, came out earlier, at 08:00 UTC. So a\n",
    "model run at 10:00 in Berlin may use the load forecast for 2 March but not its\n",
    "prices. A model never has to remember this rule: the platform applies it to\n",
    "every read.\n",
    "\n",
    "## A week, drawn\n",
    "\n",
    "Look at a whole week before you try to forecast one."
   ]
  },
  {
   "cell_type": "code",
   "id": "code-14",
   "metadata": {},
   "source": [
    "week = lab.series(\"epf.price\", \"2017-02-26T23:00:00Z\", \"2017-03-05T23:00:00Z\")\n",
    "week.tz_convert(\"Europe/Berlin\").plot(drawstyle=\"steps-post\", figsize=(10, 3),\n",
    "                                      ylabel=\"EUR/MWh\", grid=True);"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "markdown-15",
   "metadata": {},
   "source": [
    "Two patterns stand out:\n",
    "\n",
    "1. **The day has a shape.** Prices climb in the morning, dip at midday and\n",
    "   peak again in the evening, when people come home.\n",
    "2. **The week has a shape.** Saturday and Sunday are cheaper: factories and\n",
    "   offices use less power.\n",
    "\n",
    "A good forecast must know both. The next notebook builds a model on them.\n",
    "\n",
    "## Try it yourself\n",
    "\n",
    "1. Read `epf.renewables_forecast` for 2 March. When did it become known? Could\n",
    "   a model run at 08:30 UTC on 1 March use it?\n",
    "2. Draw the week of Christmas 2016, from `2016-12-18T23:00:00Z` to\n",
    "   `2016-12-25T23:00:00Z`. Which pattern breaks, and why?"
   ]
  }
 ],
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 }
}
