
[{"content":"","date":"November 15, 2026","externalUrl":null,"permalink":"/tags/apple/","section":"Tags","summary":"","title":"Apple","type":"tags"},{"content":"I really like the Apple Home app: dark tiles, a badge that lights up, no thinking about protocols. What I don\u0026rsquo;t like is being limited to what Apple decides to surface, with the only escape hatch being HomeKit bridges.\nHere\u0026rsquo;s the deal I actually want: the same experience, self-hosted, running on the Mac (or a Raspberry Pi) that already sits on my home network, talking to Home Assistant — which is already the brain of the house.\nThat turned out to be one Python file, standard library only. This post is a complete, verified walkthrough: every output below is from a real run.\nWhat we\u0026rsquo;re building # A little server that:\nReads device state from Home Assistant\u0026rsquo;s REST API (/api/states) Serves an Apple Home–style UI: toggles, dimmer sliders, thermostats, sensors, door sensors Proxies taps/sliders back to HA service calls (light.turn_on, climate.set_temperature, …) Works from any phone or laptop on the home network — no app install Here it is, running against my setup (this is a real screenshot, not a mockup):\nOn a phone it goes to the familiar two-column Apple Home layout:\nSafari → Share → Add to Home Screen and it feels like an app.\nArchitecture # iPhone / laptop / iPad Mac or Raspberry Pi Home Assistant ┌──────────────────┐ http ┌────────────────────┐ REST ┌──────────────┐ │ Safari, LAN │◄─────────►│ homeview_app.py │◄───────►│ homeassistant│ │ Add-to-Home │ :5055 │ stdlib HTTP │ :8123 │ .local │ └──────────────────┘ └────────────────────┘ └──────────────┘ │ your devices (Zigbee/Z-Wave/WiFi) No cloud, no HomeKit bridge, no dependencies to install. If the router forwards nothing and the Pi is on the LAN, this simply is not reachable from outside the house.\nPrerequisites # A Home Assistant instance on the LAN (http://homeassistant.local:8123) A long-lived access token: in Home Assistant → your avatar → Security → Long-lived access tokens → Create token. Copy it once; you won\u0026rsquo;t see it again. macOS or a Raspberry Pi with Python 3 (anything ≥3.7; zero pip installs) Test it without a smart home first: the mock # Because I want this post reproducible without physical devices, everything here runs against a ~80-line mock Home Assistant (scripts/mock_homeassistant.py in the repo). It implements exactly the two endpoints we need, with six fake entities:\n\u0026#34;\u0026#34;\u0026#34;Minimal stand-in for the Home Assistant REST API, for offline demos/tests. python mock_homeassistant.py # http://127.0.0.1:8125 \u0026#34;\u0026#34;\u0026#34; import json from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer TOKEN = \u0026#34;mock-token\u0026#34; states = { \u0026#34;light.living_room\u0026#34;: {\u0026#34;state\u0026#34;: \u0026#34;on\u0026#34;, \u0026#34;attributes\u0026#34;: {\u0026#34;friendly_name\u0026#34;: \u0026#34;Living room\u0026#34;, \u0026#34;brightness\u0026#34;: 180}}, \u0026#34;light.kitchen\u0026#34;: {\u0026#34;state\u0026#34;: \u0026#34;off\u0026#34;, \u0026#34;attributes\u0026#34;: {\u0026#34;friendly_name\u0026#34;: \u0026#34;Kitchen\u0026#34;, \u0026#34;brightness\u0026#34;: 255}}, \u0026#34;switch.office_fan\u0026#34;: {\u0026#34;state\u0026#34;: \u0026#34;off\u0026#34;, \u0026#34;attributes\u0026#34;: {\u0026#34;friendly_name\u0026#34;: \u0026#34;Office fan\u0026#34;}}, \u0026#34;binary_sensor.front_door\u0026#34;: {\u0026#34;state\u0026#34;: \u0026#34;off\u0026#34;, \u0026#34;attributes\u0026#34;: {\u0026#34;friendly_name\u0026#34;: \u0026#34;Front door\u0026#34;, \u0026#34;device_class\u0026#34;: \u0026#34;door\u0026#34;}}, \u0026#34;sensor.study_temp\u0026#34;: {\u0026#34;state\u0026#34;: \u0026#34;21.4\u0026#34;, \u0026#34;attributes\u0026#34;: {\u0026#34;friendly_name\u0026#34;: \u0026#34;Study\u0026#34;, \u0026#34;unit_of_measurement\u0026#34;: \u0026#34;°C\u0026#34;}}, \u0026#34;climate.thermostat\u0026#34;: {\u0026#34;state\u0026#34;: \u0026#34;heat\u0026#34;, \u0026#34;attributes\u0026#34;: {\u0026#34;friendly_name\u0026#34;: \u0026#34;Nest hall\u0026#34;, \u0026#34;temperature\u0026#34;: 21.5, \u0026#34;current_temperature\u0026#34;: 20.1}}, } def snapshot(): # HA state objects always carry entity_id — keep that contract. return [{\u0026#34;entity_id\u0026#34;: eid, **st} for eid, st in states.items()] def apply_service(domain, service, payload): entity_id = (payload or {}).get(\u0026#34;entity_id\u0026#34;) if not entity_id or entity_id not in states: return attrs = states[entity_id][\u0026#34;attributes\u0026#34;] if domain == \u0026#34;light\u0026#34; and service in (\u0026#34;turn_on\u0026#34;, \u0026#34;turn_off\u0026#34;): if service == \u0026#34;turn_on\u0026#34;: states[entity_id][\u0026#34;state\u0026#34;] = \u0026#34;on\u0026#34; if \u0026#34;brightness\u0026#34; in (payload or {}): attrs[\u0026#34;brightness\u0026#34;] = payload[\u0026#34;brightness\u0026#34;] else: states[entity_id][\u0026#34;state\u0026#34;] = \u0026#34;off\u0026#34; elif domain == \u0026#34;switch\u0026#34; and service in (\u0026#34;turn_on\u0026#34;, \u0026#34;turn_off\u0026#34;): states[entity_id][\u0026#34;state\u0026#34;] = \u0026#34;on\u0026#34; if service == \u0026#34;turn_on\u0026#34; else \u0026#34;off\u0026#34; elif domain == \u0026#34;climate\u0026#34; and service == \u0026#34;set_temperature\u0026#34;: attrs[\u0026#34;temperature\u0026#34;] = (payload or {}).get(\u0026#34;temperature\u0026#34;, attrs[\u0026#34;temperature\u0026#34;]) class Handler(BaseHTTPRequestHandler): def _send(self, code, obj): body = json.dumps(obj).encode() self.send_response(code) self.send_header(\u0026#34;Content-Type\u0026#34;, \u0026#34;application/json\u0026#34;) self.send_header(\u0026#34;Content-Length\u0026#34;, str(len(body))) self.end_headers() self.wfile.write(body) def do_GET(self): if self.headers.get(\u0026#34;Authorization\u0026#34;) != f\u0026#34;Bearer {TOKEN}\u0026#34;: return self._send(401, {\u0026#34;message\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) if self.path == \u0026#34;/api/states\u0026#34;: return self._send(200, snapshot()) self._send(404, {\u0026#34;message\u0026#34;: \u0026#34;Not found\u0026#34;}) def do_POST(self): if self.headers.get(\u0026#34;Authorization\u0026#34;) != f\u0026#34;Bearer {TOKEN}\u0026#34;: return self._send(401, {\u0026#34;message\u0026#34;: \u0026#34;Unauthorized\u0026#34;}) parts = self.path.strip(\u0026#34;/\u0026#34;).split(\u0026#34;/\u0026#34;) # api/services/\u0026lt;domain\u0026gt;/\u0026lt;service\u0026gt; if len(parts) == 4 and parts[0] == \u0026#34;api\u0026#34; and parts[1] == \u0026#34;services\u0026#34;: length = int(self.headers.get(\u0026#34;Content-Length\u0026#34;, 0)) payload = json.loads(self.rfile.read(length)) if length else {} apply_service(parts[2], parts[3], payload) return self._send(200, snapshot()) self._send(404, {\u0026#34;message\u0026#34;: \u0026#34;Not found\u0026#34;}) def log_message(self, *args): pass if __name__ == \u0026#34;__main__\u0026#34;: print(\u0026#34;mock Home Assistant on http://127.0.0.1:8125 — token: mock-token\u0026#34;) ThreadingHTTPServer((\u0026#34;127.0.0.1\u0026#34;, 8125), Handler).serve_forever() The app: one file, three parts # homeview_app.py has three pieces.\n1. Config and HA plumbing # import json, os, urllib.request from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer HA_URL = os.environ.get(\u0026#34;HA_URL\u0026#34;, \u0026#34;http://homeassistant.local:8123\u0026#34;).rstrip(\u0026#34;/\u0026#34;) TOKEN = os.environ.get(\u0026#34;HA_TOKEN\u0026#34;, \u0026#34;\u0026#34;) PORT = int(os.environ.get(\u0026#34;PORT\u0026#34;, \u0026#34;5055\u0026#34;)) # tile: (entity_id, type) type: toggle | dimmer | switch | sensor | climate | door TILES = [ (\u0026#34;light.living_room\u0026#34;, \u0026#34;toggle\u0026#34;), (\u0026#34;light.kitchen\u0026#34;, \u0026#34;dimmer\u0026#34;), (\u0026#34;switch.office_fan\u0026#34;, \u0026#34;switch\u0026#34;), (\u0026#34;sensor.study_temp\u0026#34;, \u0026#34;sensor\u0026#34;), (\u0026#34;climate.thermostat\u0026#34;, \u0026#34;climate\u0026#34;), (\u0026#34;binary_sensor.front_door\u0026#34;, \u0026#34;door\u0026#34;), ] API_HEADERS = {\u0026#34;Authorization\u0026#34;: f\u0026#34;Bearer {TOKEN}\u0026#34;, \u0026#34;Content-Type\u0026#34;: \u0026#34;application/json\u0026#34;} def ha_get(path): req = urllib.request.Request(f\u0026#34;{HA_URL}{path}\u0026#34;, headers=API_HEADERS) with urllib.request.urlopen(req, timeout=10) as r: return json.loads(r.read()) def ha_service(domain, service, payload): req = urllib.request.Request( f\u0026#34;{HA_URL}/api/services/{domain}/{service}\u0026#34;, data=json.dumps(payload).encode(), headers=API_HEADERS, method=\u0026#34;POST\u0026#34;) with urllib.request.urlopen(req, timeout=10) as r: return json.loads(r.read()) The tile list is the entire configuration surface — add a line, get a tile. Editing no HTML for new devices of an already-known type.\n2. The UI # One HTML string served at /. The styling does the heavy lifting for the Apple look — big radii, near-black cards, a single accent, badges instead of checkboxes:\n:root{--bg:#0b0e11;--card:#16191d;--card-on:#1d2b33;--sub:#8a9199;--on:#5ec9c2;} .grid{display:grid;grid-template-columns:repeat(2,1fr);gap:12px;max-width:520px;margin:0 auto} @media(min-width:760px){.grid{grid-template-columns:repeat(4,1fr)}} .tile{background:var(--card);border-radius:22px;padding:16px;min-height:110px; position:relative;cursor:pointer;transition:border .15s,transform .1s} .tile.on{background:var(--card-on)} .tile:active{transform:scale(.97)} /* iOS press feedback */ .tile .badge{position:absolute;top:14px;right:14px;width:26px;height:26px; border-radius:50%;background:#2a2f36} .tile.on .badge{background:var(--on);color:#08242e} The JavaScript is tiny because the state contract is tiny. It polls every 5 s, and taps are optimistic: the tile flips instantly, the service call follows:\nasync function toggle(id){ const st = find(id), domain = id.split(\u0026#39;.\u0026#39;)[0], on = st.state === \u0026#39;on\u0026#39;; st.state = on ? \u0026#39;off\u0026#39; : \u0026#39;on\u0026#39;; render(); // optimistic flip await fetch(\u0026#39;/api/services/\u0026#39; + domain + \u0026#39;/\u0026#39; + (on ? \u0026#39;turn_off\u0026#39; : \u0026#39;turn_on\u0026#39;), {method:\u0026#39;POST\u0026#39;, headers:{\u0026#39;Content-Type\u0026#39;:\u0026#39;application/json\u0026#39;}, body: JSON.stringify({entity_id: id})}); } Dimmers get a native \u0026lt;input type=range\u0026gt; that commits on release; the thermostat is two buttons firing climate.set_temperature in ±0.5° steps.\n3. The HTTP server # class Handler(BaseHTTPRequestHandler): def do_GET(self): if self.path == \u0026#34;/\u0026#34;: html = PAGE.replace(\u0026#34;__TILES__\u0026#34;, json.dumps(TILES)) return self._send(200, html.encode(), \u0026#34;text/html; charset=utf-8\u0026#34;) if self.path == \u0026#34;/api/states\u0026#34;: wanted = {e for e, _ in TILES} states = [s for s in ha_get(\u0026#34;/api/states\u0026#34;) if s[\u0026#34;entity_id\u0026#34;] in wanted] return self._send(200, json.dumps(states).encode()) self._send(404, b\u0026#39;{\u0026#34;error\u0026#34;:\u0026#34;not found\u0026#34;}\u0026#39;) def do_POST(self): length = int(self.headers.get(\u0026#34;Content-Length\u0026#34;, 0)) payload = json.loads(self.rfile.read(length)) if length else {} parts = self.path.strip(\u0026#34;/\u0026#34;).split(\u0026#34;/\u0026#34;) if len(parts) == 4 and parts[0] == \u0026#34;api\u0026#34; and parts[1] == \u0026#34;services\u0026#34;: return self._send(200, json.dumps(ha_service(parts[2], parts[3], payload)).encode()) self._send(404, b\u0026#39;{\u0026#34;error\u0026#34;:\u0026#34;not found\u0026#34;}\u0026#39;) Note len(parts) == 4 — /api/services/light/turn_off splits into four segments. My first draft checked for three and every toggle silently 404\u0026rsquo;d. A classic proxy off-by-one; the fix is in both the mock and the app.\nRun it # # terminal 1 — fake devices (or skip if you have real HA) python3 scripts/mock_homeassistant.py # terminal 2 — the dashboard HA_URL=http://127.0.0.1:8125 HA_TOKEN=mock-token PORT=5055 python3 scripts/homeview_app.py # HomeView http://192.168.0.102:5055 (HA: http://127.0.0.1:8125) Real session against the running stack — state listing:\nlight.living_room = on, light.kitchen = off, switch.office_fan = off, binary_sensor.front_door = off, sensor.study_temp = 21.4, climate.thermostat = heat Turning things off/on and dimming through the proxy (raw curl, exactly what the tiles send):\nPOST /api/services/light/turn_off {\u0026#34;entity_id\u0026#34;: \u0026#34;light.living_room\u0026#34;} POST /api/services/light/turn_on {\u0026#34;entity_id\u0026#34;: \u0026#34;light.kitchen\u0026#34;, \u0026#34;brightness\u0026#34;: 90} POST /api/services/climate/set_temperature {\u0026#34;entity_id\u0026#34;: \u0026#34;climate.thermostat\u0026#34;, \u0026#34;temperature\u0026#34;: 23} Result, re-reading state afterwards:\nlight.living_room = off 180 light.kitchen = on 90 climate.thermostat = heat 23 Toggles, dimmer levels and setpoints — the full loop works round-trip.\nMake it permanent # On a Mac — a launchd agent starts it at login:\ncp scripts/homeview_app.py ~/bin/ \u0026amp;\u0026amp; mkdir -p ~/Library/LaunchAgents cat \u0026gt; ~/Library/LaunchAgents/local.homeview.plist \u0026lt;\u0026lt; \u0026#39;EOF\u0026#39; \u0026lt;?xml version=\u0026#34;1.0\u0026#34; encoding=\u0026#34;UTF-8\u0026#34;?\u0026gt; \u0026lt;!DOCTYPE plist PUBLIC \u0026#34;-//Apple//DTD PLIST 1.0//EN\u0026#34; \u0026#34;http://www.apple.com/DTDs/PropertyList-1.0.dtd\u0026#34;\u0026gt; \u0026lt;plist version=\u0026#34;1.0\u0026#34;\u0026gt;\u0026lt;dict\u0026gt; \u0026lt;key\u0026gt;Label\u0026lt;/key\u0026gt;\u0026lt;string\u0026gt;local.homeview\u0026lt;/string\u0026gt; \u0026lt;key\u0026gt;ProgramArguments\u0026lt;/key\u0026gt;\u0026lt;array\u0026gt; \u0026lt;string\u0026gt;/usr/bin/python3\u0026lt;/string\u0026gt;\u0026lt;string\u0026gt;/Users/YOU/bin/homeview_app.py\u0026lt;/string\u0026gt; \u0026lt;/array\u0026gt; \u0026lt;key\u0026gt;EnvironmentVariables\u0026lt;/key\u0026gt;\u0026lt;dict\u0026gt; \u0026lt;key\u0026gt;HA_URL\u0026lt;/key\u0026gt;\u0026lt;string\u0026gt;http://homeassistant.local:8123\u0026lt;/string\u0026gt; \u0026lt;key\u0026gt;HA_TOKEN\u0026lt;/key\u0026gt;\u0026lt;string\u0026gt;YOUR_LONG_LIVED_TOKEN\u0026lt;/string\u0026gt; \u0026lt;/dict\u0026gt; \u0026lt;key\u0026gt;RunAtLoad\u0026lt;/key\u0026gt;\u0026lt;true/\u0026gt; \u0026lt;key\u0026gt;KeepAlive\u0026lt;/key\u0026gt;\u0026lt;true/\u0026gt; \u0026lt;/dict\u0026gt;\u0026lt;/plist\u0026gt; EOF launchctl load ~/Library/LaunchAgents/local.homeview.plist On a Raspberry Pi — systemd:\n# /etc/systemd/system/homeview.service [Unit] Description=HomeView dashboard After=network-online.target [Service] User=pi Environment=HA_URL=http://homeassistant.local:8123 Environment=HA_TOKEN=YOUR_LONG_LIVED_TOKEN ExecStart=/usr/bin/python3 /home/pi/homeview_app.py Restart=always [Install] WantedBy=multi-user.target sudo systemctl enable --now homeview The Pi bonus: it\u0026rsquo;s always on, sips ~2 % CPU with this server, and the dashboard survives router reboots because nothing external is involved.\nA token, two secrets, three rules # Never commit the token. It\u0026rsquo;s passed via environment / plist / unit file. (If you git grep HA_TOKEN and find one —rotate it immediately.) Port 5055 listens on 0.0.0.0 so phones can reach it — that also means every LAN guest can. The app proxies auth to HA, but if your guests are sketchy, bind to a non-routable interface or add a catch-all auth check in do_GET/do_POST (five lines behind an env var). No port forwarding, ever. If you need it outside the house, that\u0026rsquo;s a job for a VPN (WireGuard, Tailscale) — not for opening 5055. Bonus: inside the Home Assistant app itself # Since the dashboard is just a web page, you can embed it in the HA sidebar with a webpage card — this snippet is from my existing create_ha_dashboard.py (HassIO websocket, token auth, idempotent):\nCONFIG = { \u0026#34;version\u0026#34;: 1, \u0026#34;views\u0026#34;: [{ \u0026#34;title\u0026#34;: \u0026#34;HomeView\u0026#34;, \u0026#34;panel\u0026#34;: True, \u0026#34;cards\u0026#34;: [ {\u0026#34;type\u0026#34;: \u0026#34;webpage\u0026#34;, \u0026#34;url\u0026#34;: \u0026#34;http://192.168.0.102:5055/\u0026#34;, \u0026#34;title\u0026#34;: \u0026#34;\u0026#34;} ], }], } Then the official HA iOS app gets a HomeView panel button that renders our tiles full-screen — the best of both: native app shell, our UI inside.\nWhat to remember # The Home Assistant REST API is small: GET /api/states + POST /api/services/\u0026lt;domain\u0026gt;/\u0026lt;service\u0026gt; cover ~90 % of dashboard needs Optimistic UI updates make a 5-second-polling dashboard feel instant One stdlib file ≈ zero deploy ceremony — if it runs Python 3, it hosts this Keep entity routing in one TILES list; the UI and the proxy both read it Test protocol-adjacent code against a 60-line mock first; it found two real bugs (the path-split off-by-one and a missing entity_id) before I touched anything live Full sources: scripts/homeview_app.py, scripts/mock_homeassistant.py in the post repo.\n","date":"November 15, 2026","externalUrl":null,"permalink":"/post/apple-home-dashboard-homeassistant/","section":"Posts","summary":"I really like the Apple Home app: dark tiles, a badge that lights up, no thinking about protocols. What I don’t like is being limited to what Apple decides to surface, with the only escape hatch being HomeKit bridges.\nHere’s the deal I actually want: the same experience, self-hosted, running on the Mac (or a Raspberry Pi) that already sits on my home network, talking to Home Assistant — which is already the brain of the house.\nThat turned out to be one Python file, standard library only. This post is a complete, verified walkthrough: every output below is from a real run.\n","title":"Build an Apple Home-Style Dashboard on a Mac or Raspberry Pi (Powered by Home Assistant)","type":"post"},{"content":"","date":"November 15, 2026","externalUrl":null,"permalink":"/tags/home-assistant/","section":"Tags","summary":"","title":"Home Assistant","type":"tags"},{"content":"","date":"November 15, 2026","externalUrl":null,"permalink":"/tags/home-automation/","section":"Tags","summary":"","title":"Home Automation","type":"tags"},{"content":"","date":"November 15, 2026","externalUrl":null,"permalink":"/tags/lan/","section":"Tags","summary":"","title":"LAN","type":"tags"},{"content":"","date":"November 15, 2026","externalUrl":null,"permalink":"/post/","section":"Posts","summary":"","title":"Posts","type":"post"},{"content":"","date":"November 15, 2026","externalUrl":null,"permalink":"/tags/python/","section":"Tags","summary":"","title":"Python","type":"tags"},{"content":"Welcome to Quick Insights! Practical articles on quant analysis methods, equity alternative data, and applied AI - each with working code you can use today.\n","date":"November 15, 2026","externalUrl":null,"permalink":"/","section":"Quick Insights","summary":"Quant analysis, equity alternative data, and AI implementation techniques - short, practical, code-first insights.","title":"Quick Insights","type":"page"},{"content":"","date":"November 15, 2026","externalUrl":null,"permalink":"/tags/raspberry-pi/","section":"Tags","summary":"","title":"Raspberry Pi","type":"tags"},{"content":"","date":"November 15, 2026","externalUrl":null,"permalink":"/tags/smart-home/","section":"Tags","summary":"","title":"Smart Home","type":"tags"},{"content":"","date":"November 15, 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"November 15, 2026","externalUrl":null,"permalink":"/tags/monte-carlo/","section":"Tags","summary":"","title":"Monte Carlo","type":"tags"},{"content":"","date":"November 15, 2026","externalUrl":null,"permalink":"/tags/numpy/","section":"Tags","summary":"","title":"NumPy","type":"tags"},{"content":"","date":"November 15, 2026","externalUrl":null,"permalink":"/tags/options/","section":"Tags","summary":"","title":"Options","type":"tags"},{"content":"","date":"November 15, 2026","externalUrl":null,"permalink":"/tags/pricing/","section":"Tags","summary":"","title":"Pricing","type":"tags"},{"content":"Black-Scholes gives you a closed-form price for a vanilla European option in one line — so why bother simulating? Because the moment an option\u0026rsquo;s payoff depends on the path (Asian, lookback, barrier) or you add non-linearity (American exercise, Vol surfaces, XVA), the formulas stop existing and Monte Carlo keeps working.\nIn this post we build a complete Monte Carlo pricer from scratch in NumPy, verify it against Black-Scholes, then extend it to path-dependent payoffs and Greeks. Every number shown comes from a single run with a fixed seed (numpy.random.default_rng(42)) — you should be able to reproduce all of them.\nThe model: geometric Brownian motion # Under the risk-neutral measure the asset follows GBM:\ndS/S = r·dt + σ·dW → S(t+dt) = S(t)·exp((r − σ²/2)·dt + σ·√dt·Z) The discretization is exact in log-space, which makes the simulation a one-liner with cumsum: draw the standard normals once as a matrix, and every path exists in memory before you price anything.\nimport numpy as np from scipy.stats import norm rng = np.random.default_rng(42) S0, K, r, T, sigma = 100.0, 100.0, 0.05, 1.0, 0.20 n_steps, n_paths = 100, 100_000 dt = T / n_steps Z = rng.standard_normal((n_paths, n_steps)) log_paths = (r - 0.5 * sigma**2) * dt + sigma * np.sqrt(dt) * Z S = np.empty((n_paths, n_steps + 1)) S[:, 0] = S0 S[:, 1:] = S0 * np.exp(np.cumsum(log_paths, axis=1)) Price a European call — and verify it # MC pricing is three steps: payoff → average → discount.\nST = S[:, -1] df = np.exp(-r * T) payoff = np.maximum(ST - K, 0) mc = df * payoff.mean() stderr = df * payoff.std(ddof=1) / np.sqrt(n_paths) Then check it against the closed form:\nd1 = (np.log(S0 / K) + (r + 0.5 * sigma**2) * T) / (sigma * np.sqrt(T)) d2 = d1 - sigma * np.sqrt(T) bs = S0 * norm.cdf(d1) - K * np.exp(-r * T) * norm.cdf(d2) Output from the run:\nBlack-Scholes reference: 10.4506 Plain MC: 10.4750 (95% CI ± 0.0915) 10.4506 sits comfortably inside the Monte Carlo confidence interval. Never report a simulator output without its standard error — SE = σ/√n shrinks with √n, so 100× more paths buys you only 10× tighter error bars.\nHalve the error for free: antithetic variates # Every GBM path is driven by normals Z. Average each payoff with the payoff of the path driven by −Z and payoffs average around a common mean — the two estimation errors partially cancel:\n# Mirror ONLY the random part; keep the drift term exactly as-is D = (r - 0.5 * sigma**2) * T # total drift over T ST_anti = S0 * np.exp(2 * D - np.cumsum(log_paths, axis=1)[:, -1]) payoff_anti = 0.5 * (np.maximum(ST - K, 0) + np.maximum(ST_anti - K, 0)) mc_anti = df * payoff_anti.mean() se_anti = df * payoff_anti.std(ddof=1) / np.sqrt(n_paths) Plain MC: 10.4750 (95% CI ± 0.0915) Antithetic MC: 10.4708 (95% CI ± 0.0458) ← 50% SE reduction, same 100,000 paths Watch the classic bug here: mirror the noise −Z, not the whole log-path −L. I mirrored exp(−cumsum(log_paths)) on the first attempt and got $8.78 — silently biased, perfectly plausible-looking, and completely wrong. The drift term must stay +D in both members of the pair; only the Brownian part flips sign.\nPath-dependent payoffs — where MC earns its keep # Each of these is one line on top of the path matrix S:\n# Asian (arithmetic average strike) — payoff on the mean of the path asian = df * np.maximum(S.mean(axis=1) - K, 0).mean() # Lookback — strike is the path maximum the holder \u0026#34;would have\u0026#34; seen lookback = df * np.maximum(S.max(axis=1) - K, 0).mean() # Up-and-out barrier at B = 120 — knocked out if the barrier is ever touched B = 120.0 hit = (S \u0026gt;= B).any(axis=1) barrier = df * np.where(hit, 0, np.maximum(ST - K, 0)).mean() Asian option: 5.7481 Lookback option: 17.8882 Barrier (120 UO): 1.4197 (38.1% of paths knocked out) Three exotic prices from the same simulated path matrix. This is the real economy of Monte Carlo: simulate once, price many derivatives.\nOne subtlety worth knowing: discretely monitored barrier options are priced higher than continuously monitored ones (the barrier can be \u0026ldquo;jumped over\u0026rdquo; between grid points) — for exact continuous-monitoring prices you\u0026rsquo;d use Brownian-bridge correction. Our number is the discrete-monitoring price, which is what the code simulates.\nGreeks from the same Z # Bump-and-reprice, but with common random numbers: reuse the identical Z matrix for the bumped and unbumped runs. Without it, finite-difference Greeks are pure noise at realistic bump sizes.\nZ_fix = rng.standard_normal((n_paths, n_steps)) # one Z, reused everywhere below def mc_price(S0_=S0, sigma_=sigma, Z=Z_fix): lp = (r - 0.5 * sigma_**2) * dt + sigma_ * np.sqrt(dt) * Z ST_ = S0_ * np.exp(np.cumsum(lp, axis=1))[:, -1] return df * np.maximum(ST_ - K, 0).mean() h = 0.01 delta = (mc_price(S0_=(1 + h) * S0) - mc_price(S0_=(1 - h) * S0)) / (2 * h * S0) vega = (mc_price(sigma_=(1 + h) * sigma) - mc_price(sigma_=(1 - h) * sigma)) / (2 * h * sigma) Delta 0.6355 (Black-Scholes: 0.6368) Vega 37.3712 (Black-Scholes: 37.5240) Both land within a hair of the closed form — the residual gap is finite-difference error plus MC error, and shrinking h further eventually just adds noise (another reason to always know your SE).\nAmerican options: LSM # European-style payoff tricks don\u0026rsquo;t capture the right to exercise early. Longstaff–Schwartz (Least Squares Monte Carlo) approaches it by regressing the continuation value on the state at each time step, walking backwards:\nfrom numpy.linalg import lstsq def lsm_put(paths, strike, rate, maturity, steps): dt_ = maturity / steps cf = np.maximum(strike - paths[:, -1], 0.0) # payoff at expiry t_cf = np.full(len(paths), steps, dtype=int) # time index of each cashflow for t in range(steps - 1, 0, -1): itm = np.where(paths[:, t] \u0026lt; strike)[0] if len(itm) == 0: continue disc = np.exp(-rate * dt_ * (t_cf[itm] - t)) # discount each cashflow to time t Y = disc * cf[itm] s = paths[itm, t] / strike # normalized state → well-conditioned X = np.vstack([np.ones(len(itm)), s, s**2]).T beta = lstsq(X, Y, rcond=None)[0] continuation = beta[0] + beta[1] * s + beta[2] * s**2 exercise = strike - paths[itm, t] early = continuation \u0026lt; exercise idx = itm[early] # commit to early exercise cf[idx], t_cf[idx] = exercise[early], t return (np.exp(-rate * dt_ * t_cf) * cf).mean() Compared to the naive \u0026ldquo;keep one values array\u0026rdquo; version you\u0026rsquo;ll find in older tutorials, tracking each path\u0026rsquo;s cashflow time (t_cf) matters: a path that skips several steps before exercising must be discounted from its actual exercise date, not by one step. Running it (200,000 paths, 100 steps):\nEuropean put (closed form): 5.5735 American put (LSM): 6.0581 ← early-exercise premium ≈ 0.48 Two honesty notes from practitioners: LSM gives a lower bound on the true American value (early exercise is under-detected through regression noise), and the regression basis / ITM filter matter more than people expect — the s = S/K normalization above isn\u0026rsquo;t cosmetic, it keeps the linear algebra well-conditioned.\nBonus: portfolio VaR for free # The same simulator answers risk questions in one line (this is essentially the Monte Carlo VaR from the Basel/FRTB toolbox):\nreturns = rng.multivariate_normal(mean_returns, cov_matrix, 100_000) # cross-asset shocks portfolio_returns = returns @ weights VaR_95 = -np.percentile(portfolio_returns, 5) ES_95 = -portfolio_returns[portfolio_returns \u0026lt; np.percentile(portfolio_returns, 5)].mean() ES (expected shortfall, the average loss beyond the VaR threshold) is what FRTB moved to for internal-models capital — VaR tells you where the tail starts, ES tells you how deep it goes.\nWhat to remember # Simulate in log-space with cumsum — vectorized, exact per step, one matrix holds every path Always report SE (± 1.96·σ/√n); error shrinks like √n, so precision is expensive — variance reduction is cheaper Antithetic variates: mirror −Z, keep the drift; ~50% SE cut here for free Common random numbers for finite-difference Greeks — reuse the same Z Verify against a closed form whenever one exists — it\u0026rsquo;s the cheapest bug detector you have Discrete ≠ continuous monitoring for barriers; LSM gives a lower bound for Americans References # Boyle, P. (1977). Options: A Monte Carlo Approach — the original Longstaff, F. \u0026amp; Schwartz, E. (2001). Valuing American Options by Simulation: A Simple Least-Squares Approach Glasserman, P. Monte Carlo Methods in Financial Engineering — Paul Glasserman [Amazon] Hull, J.C. Options, Futures, and Other Derivatives — John C. Hull [Amazon] ","date":"November 15, 2026","externalUrl":null,"permalink":"/post/monte_carlo_option_pricing/","section":"Posts","summary":"Black-Scholes gives you a closed-form price for a vanilla European option in one line — so why bother simulating? Because the moment an option’s payoff depends on the path (Asian, lookback, barrier) or you add non-linearity (American exercise, Vol surfaces, XVA), the formulas stop existing and Monte Carlo keeps working.\nIn this post we build a complete Monte Carlo pricer from scratch in NumPy, verify it against Black-Scholes, then extend it to path-dependent payoffs and Greeks. Every number shown comes from a single run with a fixed seed (numpy.random.default_rng(42)) — you should be able to reproduce all of them.\n","title":"Pricing Options with Monte Carlo in NumPy — Verified Against Black-Scholes","type":"post"},{"content":"","date":"November 15, 2026","externalUrl":null,"permalink":"/tags/quantitative-finance/","section":"Tags","summary":"","title":"Quantitative Finance","type":"tags"},{"content":"Happy to discuss questions, ideas, or collaboration around anything on this site — quant methods, data engineering, forecasting, or applied AI.\nLinkedIn: linkedin.com/in/yauheni-yakauleu Or use the form below — it goes straight to me. Email: Message: Send ","date":"November 15, 2026","externalUrl":null,"permalink":"/contact/","section":"Quick Insights","summary":"Happy to discuss questions, ideas, or collaboration","title":"Contact","type":"page"},{"content":"","date":"March 12, 2025","externalUrl":null,"permalink":"/tags/credit-card-sales/","section":"Tags","summary":"","title":"Credit Card Sales","type":"tags"},{"content":"","date":"March 12, 2025","externalUrl":null,"permalink":"/tags/holidays/","section":"Tags","summary":"","title":"Holidays","type":"tags"},{"content":"","date":"March 12, 2025","externalUrl":null,"permalink":"/tags/machine-learning/","section":"Tags","summary":"","title":"Machine Learning","type":"tags"},{"content":" Predicting Credit Card Sales: A Comprehensive Guide to Quarterly Forecasting # Understanding the Key Components of Credit Card Sales Time Series # Credit card sales data typically exhibits several important patterns that must be accounted for in your forecasting model:\nTrend Component: The long-term direction of sales (growth or decline) Seasonal Components: Weekly Seasonality: Different spending patterns by day of week Monthly Seasonality: Monthly payment cycles and salary effects Quarterly Seasonality: Business cycles and quarterly reporting effects Annual Seasonality: Yearly patterns like holiday shopping seasons Holiday Effects: Specific spikes or drops around holidays Special Events: Sales, promotions, or other company-specific events Data Preparation and Exploration # Before building your model, you should:\nGather historical data: Collect at least 2-3 years of daily credit card sales data\nPerform exploratory data analysis:\nPlot the time series to identify obvious patterns Create day-of-week, month, and quarter aggregations to understand seasonality Calculate year-over-year growth rates to understand the trend Identify outliers and anomalies Create calendar features:\nDay of week (1-7) Day of month (1-31) Month of year (1-12) Quarter (1-4) Is weekend (binary) Is holiday (binary) Days before/after major holidays Modeling Approaches # 1. Facebook Prophet # Prophet is particularly well-suited for credit card sales forecasting because it handles multiple seasonalities and holiday effects out of the box.\nfrom prophet import Prophet import pandas as pd # Prepare data in Prophet format (ds for dates, y for values) df = pd.DataFrame({ \u0026#39;ds\u0026#39;: date_column, \u0026#39;y\u0026#39;: sales_column }) # Create holiday dataframe holidays = pd.DataFrame({ \u0026#39;holiday\u0026#39;: [\u0026#39;Black Friday\u0026#39;, \u0026#39;Black Friday\u0026#39;, ...], \u0026#39;ds\u0026#39;: pd.to_datetime([\u0026#39;2022-11-25\u0026#39;, \u0026#39;2023-11-24\u0026#39;, ...]), \u0026#39;lower_window\u0026#39;: [-1, -1, ...], # Days before holiday to include \u0026#39;upper_window\u0026#39;: [1, 1, ...] # Days after holiday to include }) # Initialize model with weekly and yearly seasonality model = Prophet( yearly_seasonality=True, weekly_seasonality=True, daily_seasonality=False, # Enable if you have intraday data holidays=holidays, seasonality_mode=\u0026#39;multiplicative\u0026#39;, # Use if holiday effects scale with trend changepoint_prior_scale=0.05, # Controls flexibility of trend holidays_prior_scale=10.0 # Controls flexibility of holiday effects ) # Add quarterly seasonality model.add_seasonality( name=\u0026#39;quarterly\u0026#39;, period=365.25/4, fourier_order=5 ) # Add custom regressors for special events or promotions if \u0026#39;promotion\u0026#39; in df.columns: model.add_regressor(\u0026#39;promotion\u0026#39;) # Fit model model.fit(df) # Create future dataframe for prediction (e.g., next quarter) future = model.make_future_dataframe(periods=90) # 90 days for a quarter # Add any regressor values to future dataframe if \u0026#39;promotion\u0026#39; in df.columns: future[\u0026#39;promotion\u0026#39;] = future_promotion_values # Make prediction forecast = model.predict(future) # Aggregate to quarterly level quarterly_forecast = forecast.set_index(\u0026#39;ds\u0026#39;).resample(\u0026#39;Q\u0026#39;)[\u0026#39;yhat\u0026#39;].sum() 2. SARIMA (Seasonal ARIMA) with External Regressors # SARIMA models can capture multiple seasonal patterns and can be extended with external regressors for holidays.\nimport pandas as pd import numpy as np from statsmodels.tsa.statespace.sarimax import SARIMAX # Create dummy variables for days of week for i in range(7): df[f\u0026#39;dow_{i}\u0026#39;] = (df[\u0026#39;date\u0026#39;].dt.dayofweek == i).astype(int) # Create dummy variables for months for i in range(1, 13): df[f\u0026#39;month_{i}\u0026#39;] = (df[\u0026#39;date\u0026#39;].dt.month == i).astype(int) # Create holiday dummies holidays = [\u0026#39;Christmas\u0026#39;, \u0026#39;New_Year\u0026#39;, \u0026#39;Thanksgiving\u0026#39;, ...] for holiday in holidays: df[holiday] = 0 # Initialize # Set to 1 for holiday dates (and potentially days before/after) # ... # Fit SARIMA model with exogenous variables model = SARIMAX( df[\u0026#39;sales\u0026#39;], exog=df[[\u0026#39;dow_1\u0026#39;, \u0026#39;dow_2\u0026#39;, ..., \u0026#39;month_1\u0026#39;, ..., \u0026#39;Christmas\u0026#39;, ...]], order=(1, 1, 1), # Non-seasonal part (p, d, q) seasonal_order=(1, 1, 1, 7), # Seasonal part (P, D, Q, s) - weekly ) results = model.fit() # Create future exogenous variables future_exog = pd.DataFrame() # Fill with future day of week, month, holiday values # ... # Forecast forecast = results.forecast(steps=90, exog=future_exog) # 90 days for a quarter 3. Machine Learning Approach with Feature Engineering # For more complex patterns, a gradient boosting approach with extensive feature engineering can be effective.\nimport pandas as pd import numpy as np from lightgbm import LGBMRegressor from sklearn.model_selection import TimeSeriesSplit # Feature engineering def create_features(df): df = df.copy() # Date features df[\u0026#39;dayofweek\u0026#39;] = df[\u0026#39;date\u0026#39;].dt.dayofweek df[\u0026#39;month\u0026#39;] = df[\u0026#39;date\u0026#39;].dt.month df[\u0026#39;year\u0026#39;] = df[\u0026#39;date\u0026#39;].dt.year df[\u0026#39;dayofyear\u0026#39;] = df[\u0026#39;date\u0026#39;].dt.dayofyear df[\u0026#39;quarter\u0026#39;] = df[\u0026#39;date\u0026#39;].dt.quarter df[\u0026#39;is_weekend\u0026#39;] = df[\u0026#39;dayofweek\u0026#39;].isin([5, 6]).astype(int) # Lag features (previous day, week, month, quarter) df[\u0026#39;sales_lag1\u0026#39;] = df[\u0026#39;sales\u0026#39;].shift(1) df[\u0026#39;sales_lag7\u0026#39;] = df[\u0026#39;sales\u0026#39;].shift(7) df[\u0026#39;sales_lag30\u0026#39;] = df[\u0026#39;sales\u0026#39;].shift(30) df[\u0026#39;sales_lag90\u0026#39;] = df[\u0026#39;sales\u0026#39;].shift(90) # Rolling window features df[\u0026#39;sales_rolling_7d_mean\u0026#39;] = df[\u0026#39;sales\u0026#39;].rolling(7).mean().shift(1) df[\u0026#39;sales_rolling_30d_mean\u0026#39;] = df[\u0026#39;sales\u0026#39;].rolling(30).mean().shift(1) # Holiday features (one-hot encoded) # ... # Days before/after major holidays # ... return df # Prepare data features = create_features(df) features = features.dropna() # Drop rows with NaN from lag features # Define features and target X = features.drop([\u0026#39;date\u0026#39;, \u0026#39;sales\u0026#39;], axis=1) y = features[\u0026#39;sales\u0026#39;] # Time series cross-validation tscv = TimeSeriesSplit(n_splits=5) # Train model model = LGBMRegressor( n_estimators=1000, learning_rate=0.05, max_depth=5 ) model.fit(X, y) # Create future features for prediction future_features = create_features(future_df) future_X = future_features.drop([\u0026#39;date\u0026#39;, \u0026#39;sales\u0026#39;], axis=1) # Predict predictions = model.predict(future_X) # Aggregate to quarterly level future_df[\u0026#39;predictions\u0026#39;] = predictions quarterly_forecast = future_df.set_index(\u0026#39;date\u0026#39;).resample(\u0026#39;Q\u0026#39;)[\u0026#39;predictions\u0026#39;].sum() Handling Specific Challenges # 1. Handling Holidays # Holidays have a significant impact on credit card sales and require special attention:\nCreate a comprehensive holiday calendar: Include not just official holidays but also shopping events (Black Friday, Cyber Monday) Account for holiday shifts: Some holidays fall on different dates each year (Easter, Thanksgiving) Consider pre-holiday and post-holiday effects: Shopping often increases before holidays and decreases after Differentiate holiday types: Some holidays increase spending (Christmas), while others may decrease it (Labor Day) 2. Handling Day-of-Week Effects # Day-of-week patterns are crucial for credit card sales:\nWeekday vs. weekend spending: Consumer behavior differs significantly Payday effects: Spending often increases around common paydays (1st, 15th, last day of month) Different day-of-week patterns by industry: Restaurants peak on weekends, while business services peak mid-week 3. Handling Seasonality # Multiple seasonal patterns affect credit card sales:\nWeekly seasonality: Capture with 7-day cycles Monthly seasonality: Capture with 30-day cycles (bill payment patterns) Quarterly seasonality: Capture with 90-day cycles (business reporting cycles) Annual seasonality: Capture with 365-day cycles (holiday shopping, summer vacation) 4. Handling Special Events # Company-specific events can significantly impact sales:\nMarketing campaigns: Include timing and scale of campaigns as features Promotions and discounts: Include as binary or continuous features Product launches: Include as binary features with potential decay effects Evaluation and Refinement # To ensure your model is accurate for quarterly forecasting:\nEvaluate at multiple levels:\nDaily accuracy (MAPE, MAE, RMSE) Weekly aggregation accuracy Monthly aggregation accuracy Quarterly aggregation accuracy Use time series cross-validation:\nTrain on historical data Test on one quarter Move forward one quarter and repeat Analyze errors by segment:\nBy day of week By month By holiday/non-holiday By promotion/non-promotion periods Refine the model iteratively:\nAdd features for periods with high error Adjust holiday windows Tune model hyperparameters Implementation Recommendations # For your specific goal of predicting full quarter sales:\nStart with Prophet for its ability to handle multiple seasonalities and holidays Add custom quarterly seasonality to capture business cycle effects Create a comprehensive holiday calendar specific to your business Include day-of-week effects explicitly Add external regressors for company-specific events and promotions Aggregate daily predictions to quarterly level for final forecasts Validate on historical quarters to ensure accuracy at the quarterly level Monitoring and Updating # Once deployed, your model should be:\nRegularly updated with new data (at least monthly) Monitored for drift in accuracy Adjusted for new patterns that emerge Supplemented with human judgment for unexpected events Potential Improvements # Incorporate Real-time Data: Use real-time transaction data to improve the timeliness and accuracy of forecasts. Leverage Advanced Machine Learning Techniques: Explore deep learning models like LSTM for capturing complex temporal dependencies. Integrate External Economic Indicators: Include macroeconomic variables such as GDP growth, unemployment rates, and consumer confidence indices to enhance model robustness. Enhance Feature Engineering: Continuously refine feature sets based on emerging patterns and business insights. Implement Automated Model Selection: Use automated machine learning (AutoML) tools to streamline model selection and hyperparameter tuning. Develop a Feedback Loop: Establish a feedback mechanism to incorporate forecast errors and improve model accuracy over time. By following this comprehensive approach, you\u0026rsquo;ll be able to build a robust model for predicting credit card sales that accounts for seasonality, holidays, and day-of-week effects, ultimately providing accurate quarterly sales forecasts for your company.\n","date":"March 12, 2025","externalUrl":null,"permalink":"/post/credit_card_sales_forecasting/","section":"Posts","summary":"Learn how to build a robust model for predicting credit card sales with a focus on handling seasonality, holidays, and day-of-week effects to achieve accurate quarterly sales forecasts.","title":"Predicting Credit Card Sales: A Comprehensive Guide to Quarterly Forecasting","type":"post"},{"content":"","date":"March 12, 2025","externalUrl":null,"permalink":"/tags/prophet/","section":"Tags","summary":"","title":"Prophet","type":"tags"},{"content":"","date":"March 12, 2025","externalUrl":null,"permalink":"/tags/quarterly-forecasting/","section":"Tags","summary":"","title":"Quarterly Forecasting","type":"tags"},{"content":"","date":"March 12, 2025","externalUrl":null,"permalink":"/tags/sarima/","section":"Tags","summary":"","title":"SARIMA","type":"tags"},{"content":"","date":"March 12, 2025","externalUrl":null,"permalink":"/tags/seasonality/","section":"Tags","summary":"","title":"Seasonality","type":"tags"},{"content":"","date":"March 12, 2025","externalUrl":null,"permalink":"/tags/time-series-forecasting/","section":"Tags","summary":"","title":"Time Series Forecasting","type":"tags"},{"content":"Automatically Load Code in Jupyter Notebooks Using the IPython Startup Folder # If you frequently work with Jupyter notebooks, you\u0026rsquo;ve probably found yourself writing the same import statements, utility functions, or configuration settings at the beginning of each notebook. This repetitive task can be eliminated by taking advantage of IPython\u0026rsquo;s startup folder functionality, which allows you to automatically load code every time you start a new notebook.\nIn this guide, we\u0026rsquo;ll explore how to set up and use the IPython startup folder in both Windows and Unix-based systems (macOS/Linux).\nWhat Is the IPython Startup Folder? # The IPython startup folder is a directory where you can place Python scripts that will be executed automatically when you start an IPython kernel (which is what Jupyter notebooks use). This means any code, imports, or functions defined in these scripts will be available in all your notebooks without having to re-import or redefine them.\nThe startup folder is typically located at:\nUnix (Linux/macOS): ~/.ipython/profile_default/startup/ Windows: C:\\Users\\YourUsername\\.ipython\\profile_default\\startup\\ Benefits of Using the Startup Folder # Save time: No need to repeatedly write common imports and functions Standardize environments: Ensure all your notebooks have the same basic setup Maintain consistency: Keep your plotting styles, pandas display options, and other configurations consistent Reduce clutter: Keep your notebooks clean and focused on the actual analysis Example Startup File # Here\u0026rsquo;s a simple example of what you might put in your startup file:\n# 00-imports.py print(\u0026#34;Loading common imports...\u0026#34;) # Data manipulation import numpy as np import pandas as pd # Visualization import matplotlib.pyplot as plt import seaborn as sns # Set matplotlib to inline mode for Jupyter %matplotlib inline print(\u0026#34;Imports loaded!\u0026#34;) This script would automatically load these libraries every time you start a new Jupyter notebook, saving you from having to write these import statements each time.\nFor more detailed information, check out the IPython documentation.\n","date":"March 11, 2025","externalUrl":null,"permalink":"/post/jupyter-ipython-startup-folder-guide/","section":"Posts","summary":"Automatically Load Code in Jupyter Notebooks Using the IPython Startup Folder # If you frequently work with Jupyter notebooks, you’ve probably found yourself writing the same import statements, utility functions, or configuration settings at the beginning of each notebook. This repetitive task can be eliminated by taking advantage of IPython’s startup folder functionality, which allows you to automatically load code every time you start a new notebook.\nIn this guide, we’ll explore how to set up and use the IPython startup folder in both Windows and Unix-based systems (macOS/Linux).\n","title":"Automatically Load Code in Jupyter Notebooks Using the IPython Startup Folder","type":"post"},{"content":"","date":"March 11, 2025","externalUrl":null,"permalink":"/tags/beginners-guide/","section":"Tags","summary":"","title":"Beginner's Guide","type":"tags"},{"content":"","date":"March 11, 2025","externalUrl":null,"permalink":"/tags/data-engineering/","section":"Tags","summary":"","title":"Data Engineering","type":"tags"},{"content":"","date":"March 11, 2025","externalUrl":null,"permalink":"/tags/data-science/","section":"Tags","summary":"","title":"Data Science","type":"tags"},{"content":"","date":"March 11, 2025","externalUrl":null,"permalink":"/tags/duckdb/","section":"Tags","summary":"","title":"DuckDB","type":"tags"},{"content":"","date":"March 11, 2025","externalUrl":null,"permalink":"/tags/etl/","section":"Tags","summary":"","title":"ETL","type":"tags"},{"content":" Getting Started with Hugo: A Beginner\u0026rsquo;s Guide # Are you looking to create a fast, secure, and flexible website or blog? Look no further than Hugo, one of the most popular static site generators on the market. With its ease of use, extensive community support, and incredible performance, Hugo is an excellent choice for beginners and experienced developers alike. In this comprehensive guide, we\u0026rsquo;ll take you by the hand and walk you through the process of getting started with Hugo, from setting up your first project to deploying your site live.\nWhat is Hugo and Why Should You Use It? # Before diving into the nitty-gritty of using Hugo, let\u0026rsquo;s first understand what Hugo is and why it\u0026rsquo;s a great choice for your next web project. Hugo is an open-source static site generator (SSG) that allows you to build fast, secure, and scalable websites. Unlike traditional content management systems (CMS) like WordPress or Drupal, Hugo generates pre-built HTML files for your website, which can then be hosted on any server or CDN.\nHere are some key benefits of using Hugo:\nSpeed: Hugo sites load incredibly fast due to their static nature. Security: With no database or server-side processing, the attack surface is significantly reduced. Flexibility: Hugo supports a wide range of templates and markup languages, giving you complete control over your site\u0026rsquo;s design and structure. Setting Up Your First Hugo Project # Now that we\u0026rsquo;ve covered the basics of what Hugo is and why it\u0026rsquo;s a great choice, let\u0026rsquo;s get started with setting up our first project. Here are the steps to follow:\nInstall Hugo: The first step is to install Hugo on your computer. You can do this by downloading the latest version from the official Hugo website or by using a package manager like Homebrew (on macOS) or Chocolatey (on Windows). Create a New Site: Once Hugo is installed, you can create a new site by running the command hugo new site in your terminal. This will prompt you to enter a name for your site. Choose a Theme: Hugo comes with a variety of pre-built themes that you can use as a starting point for your site. You can browse the official Hugo theme repository to find one that suits your needs. Some popular Hugo themes for beginners include:\nAnanke: A simple, responsive theme great for blogs and personal sites. Minos: A minimalistic theme with a focus on typography and readability. Tania: A clean and modern theme suitable for business or portfolio sites. Building and Customizing Your Hugo Site # With your site set up and a theme chosen, it\u0026rsquo;s time to start building and customizing your content. Here are some tips to keep in mind:\nContent is King: In Hugo, content is stored in Markdown files within the content directory of your project. You can create new pages by adding new Markdown files. Front Matter: Each Markdown file can include front matter, which is metadata about the page such as its title, date, and categories. Templates and Partial: Hugo uses the Go template engine to render your site\u0026rsquo;s HTML. You can customize the look and feel of your site by editing the templates and partials within your theme. Some practical examples of customizing your Hugo site include:\nAdding a custom header or footer to your site. Creating a blog section with categorized posts. Integrating social media links into your site\u0026rsquo;s template. Deploying Your Hugo Site # The final step in getting started with Hugo is deploying your site live. Here are some popular options for hosting your Hugo site:\nNetlify: A platform that offers free hosting, SSL certificates, and continuous integration for static sites. Vercel: A cloud platform for hosting static sites, with features like serverless functions and edge networking. GitHub Pages: A free service offered by GitHub for hosting static sites directly from your repository. To deploy your site, you\u0026rsquo;ll typically need to:\nBuild your site using the command hugo in your terminal. Push your code to a Git repository like GitHub or GitLab. Configure your hosting platform to pull from your repository and serve your static files. Conclusion # Getting started with Hugo is easier than you might think, especially with the help of this beginner\u0026rsquo;s guide. From setting up your first project to deploying your site live, we\u0026rsquo;ve covered all the essential steps to get you up and running with Hugo. Whether you\u0026rsquo;re a seasoned developer or just starting out, Hugo offers a powerful and flexible way to build fast, secure, and scalable websites.\nSo why not give Hugo a try today? Start building your next web project with Hugo and experience the difference for yourself. With its extensive community support, rich ecosystem of themes and plugins, and incredible performance, we\u0026rsquo;re confident you\u0026rsquo;ll love using Hugo as much as we do!\n","date":"March 11, 2025","externalUrl":null,"permalink":"/posts/2025-03-11-getting-started-with-hugo-a-beginners-guide/","section":"Posts","summary":"Get started with Hugo, a popular static site generator, and learn how to create fast, secure, and flexible websites or blogs. This beginner’s guide provides a comprehensive introduction to using Hugo.","title":"Getting Started with Hugo: A Beginner's Guide","type":"posts"},{"content":"","date":"March 11, 2025","externalUrl":null,"permalink":"/tags/hugo/","section":"Tags","summary":"","title":"Hugo","type":"tags"},{"content":"","date":"March 11, 2025","externalUrl":null,"permalink":"/tags/ipython/","section":"Tags","summary":"","title":"IPython","type":"tags"},{"content":"","date":"March 11, 2025","externalUrl":null,"permalink":"/tags/jupyter/","section":"Tags","summary":"","title":"Jupyter","type":"tags"},{"content":"","date":"March 11, 2025","externalUrl":null,"permalink":"/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"","date":"March 11, 2025","externalUrl":null,"permalink":"/tags/productivity/","section":"Tags","summary":"","title":"Productivity","type":"tags"},{"content":"","date":"March 11, 2025","externalUrl":null,"permalink":"/tags/static-site-generators/","section":"Tags","summary":"","title":"Static Site Generators","type":"tags"},{"content":" Introduction to DuckDB # DuckDB has emerged as a game-changer in the data processing landscape. It\u0026rsquo;s an in-process SQL OLAP database management system, similar to SQLite but optimized for analytical queries rather than transactional workloads. What makes DuckDB particularly attractive is its remarkable performance for data analysis tasks combined with its simplicity - no server setup, no complex configuration, just a library you can import into your Python scripts or notebooks.\nIn this guide, we\u0026rsquo;ll explore how to use DuckDB effectively in both Jupyter Notebooks and Python scripts for Extract, Transform, Load (ETL) operations. We\u0026rsquo;ll cover installation, basic operations, and show practical examples for both environments.\nWhy Choose DuckDB for ETL? # Before diving into the how-to, let\u0026rsquo;s understand why DuckDB is worth considering for your ETL workflows:\nExceptional Performance: DuckDB can process large datasets at speeds comparable to (or even faster than) many distributed systems for single-node workloads. Simplicity: No separate server process, no complex setup - just import and use. SQL Interface: Familiar SQL syntax makes it accessible for those already comfortable with SQL. Seamless Integration: Works directly with Pandas DataFrames, Parquet files, CSV, and other common data formats. Column-oriented Storage: Optimized for analytical queries common in ETL processes. Installing DuckDB # Let\u0026rsquo;s start with installation. DuckDB is available via pip:\npip install duckdb For Jupyter integration, you might also want to install the Jupyter extension:\npip install duckdb-jupyter Using DuckDB in Jupyter Notebook # Jupyter Notebooks provide an interactive environment perfect for data exploration and prototyping ETL processes. Here\u0026rsquo;s how to integrate DuckDB into your notebook workflow:\nBasic Setup and Queries # import duckdb import pandas as pd import numpy as np # Create a connection conn = duckdb.connect(database=\u0026#39;:memory:\u0026#39;) # In-memory database # Create some sample data df = pd.DataFrame({ \u0026#39;id\u0026#39;: range(1, 1001), \u0026#39;value\u0026#39;: np.random.randn(1000), \u0026#39;category\u0026#39;: np.random.choice([\u0026#39;A\u0026#39;, \u0026#39;B\u0026#39;, \u0026#39;C\u0026#39;, \u0026#39;D\u0026#39;], 1000) }) # Query the DataFrame directly with SQL result = conn.execute(\u0026#34;\u0026#34;\u0026#34; SELECT category, COUNT(*) as count, AVG(value) as avg_value, SUM(value) as sum_value FROM df GROUP BY category ORDER BY count DESC \u0026#34;\u0026#34;\u0026#34;).fetchdf() result Loading and Processing Data Files # DuckDB excels at processing data files directly. Here\u0026rsquo;s an example loading and querying a CSV file:\n# Query a CSV file directly (without loading it into memory first) conn.execute(\u0026#34;CREATE TABLE sales AS SELECT * FROM read_csv_auto(\u0026#39;sales_data.csv\u0026#39;)\u0026#34;) # Execute analytical query result = conn.execute(\u0026#34;\u0026#34;\u0026#34; SELECT date_trunc(\u0026#39;month\u0026#39;, order_date) as month, region, SUM(sales_amount) as total_sales, COUNT(DISTINCT customer_id) as unique_customers FROM sales WHERE order_date \u0026gt;= \u0026#39;2022-01-01\u0026#39; GROUP BY month, region ORDER BY month, total_sales DESC \u0026#34;\u0026#34;\u0026#34;).fetchdf() # Result is a pandas DataFrame result.head() Interactive Visual Analysis # Combine DuckDB with visualization libraries for powerful visual analysis:\nimport matplotlib.pyplot as plt import seaborn as sns # Execute query and get results monthly_sales = conn.execute(\u0026#34;\u0026#34;\u0026#34; SELECT date_trunc(\u0026#39;month\u0026#39;, order_date) as month, SUM(sales_amount) as total_sales FROM sales GROUP BY month ORDER BY month \u0026#34;\u0026#34;\u0026#34;).fetchdf() # Plot results plt.figure(figsize=(12, 6)) sns.lineplot(data=monthly_sales, x=\u0026#39;month\u0026#39;, y=\u0026#39;total_sales\u0026#39;) plt.title(\u0026#39;Monthly Sales Trend\u0026#39;) plt.xticks(rotation=45) plt.tight_layout() plt.show() Using DuckDB in Python Scripts for ETL # While Jupyter is great for exploration, production ETL processes often run as standalone Python scripts. Here\u0026rsquo;s how to use DuckDB effectively in this context:\nCreating a Simple ETL Pipeline # import duckdb import os from datetime import datetime # Configuration input_dir = \u0026#39;/path/to/input\u0026#39; output_dir = \u0026#39;/path/to/output\u0026#39; log_file = \u0026#39;etl_log.txt\u0026#39; def log_message(message): \u0026#34;\u0026#34;\u0026#34;Write a log message with timestamp\u0026#34;\u0026#34;\u0026#34; timestamp = datetime.now().strftime(\u0026#34;%Y-%m-%d %H:%M:%S\u0026#34;) with open(log_file, \u0026#39;a\u0026#39;) as f: f.write(f\u0026#34;{timestamp} - {message}\\n\u0026#34;) def main(): # Log start log_message(\u0026#34;ETL process started\u0026#34;) try: # Create connection conn = duckdb.connect(\u0026#39;etl_process.db\u0026#39;) # Extract: Load data from multiple CSV files log_message(\u0026#34;Extracting data from CSV files\u0026#34;) conn.execute(\u0026#34;\u0026#34;\u0026#34; CREATE OR REPLACE TABLE raw_data AS SELECT * FROM read_csv_auto( glob(\u0026#39;{}/sales_*.csv\u0026#39;) ) \u0026#34;\u0026#34;\u0026#34;.format(input_dir)) # Transform: Clean and aggregate data log_message(\u0026#34;Transforming data\u0026#34;) conn.execute(\u0026#34;\u0026#34;\u0026#34; CREATE OR REPLACE TABLE transformed_data AS SELECT date_trunc(\u0026#39;month\u0026#39;, order_date) as month, product_category, region, SUM(CASE WHEN order_status = \u0026#39;completed\u0026#39; THEN amount ELSE 0 END) as completed_sales, COUNT(DISTINCT customer_id) as customer_count FROM raw_data WHERE order_date \u0026gt;= \u0026#39;2022-01-01\u0026#39; AND amount \u0026gt; 0 GROUP BY month, product_category, region \u0026#34;\u0026#34;\u0026#34;) # Load: Export to Parquet for downstream use output_file = os.path.join(output_dir, f\u0026#34;sales_summary_{datetime.now().strftime(\u0026#39;%Y%m%d\u0026#39;)}.parquet\u0026#34;) log_message(f\u0026#34;Loading data to {output_file}\u0026#34;) conn.execute(f\u0026#34;\u0026#34;\u0026#34; COPY transformed_data TO \u0026#39;{output_file}\u0026#39; (FORMAT PARQUET) \u0026#34;\u0026#34;\u0026#34;) # Get and log record counts raw_count = conn.execute(\u0026#34;SELECT COUNT(*) FROM raw_data\u0026#34;).fetchone()[0] transformed_count = conn.execute(\u0026#34;SELECT COUNT(*) FROM transformed_data\u0026#34;).fetchone()[0] log_message(f\u0026#34;Processed {raw_count} raw records into {transformed_count} aggregated records\u0026#34;) log_message(\u0026#34;ETL process completed successfully\u0026#34;) except Exception as e: log_message(f\u0026#34;Error in ETL process: {str(e)}\u0026#34;) raise finally: # Close connection conn.close() if __name__ == \u0026#34;__main__\u0026#34;: main() Handling Large Datasets Efficiently # For larger datasets, you can leverage DuckDB\u0026rsquo;s efficient processing capabilities:\nimport duckdb import os import argparse from datetime import datetime def process_large_dataset(input_files, output_dir, partition_by=None): \u0026#34;\u0026#34;\u0026#34;Process large datasets with optional partitioning\u0026#34;\u0026#34;\u0026#34; conn = duckdb.connect(\u0026#39;:memory:\u0026#39;) # Register input files (could be CSV, Parquet, etc.) file_list_str = \u0026#34;\u0026#39;, \u0026#39;\u0026#34;.join(input_files) # Load data with parallel processing conn.execute(f\u0026#34;\u0026#34;\u0026#34; CREATE OR REPLACE TABLE source_data AS SELECT * FROM read_parquet(\u0026#39;{file_list_str}\u0026#39;) \u0026#34;\u0026#34;\u0026#34;) # Perform transformations conn.execute(\u0026#34;\u0026#34;\u0026#34; CREATE OR REPLACE TABLE transformed_data AS SELECT /* Select your transformed columns here */ date_part(\u0026#39;year\u0026#39;, event_timestamp) AS year, date_part(\u0026#39;month\u0026#39;, event_timestamp) AS month, user_id, SUM(IF(event_type = \u0026#39;purchase\u0026#39;, amount, 0)) AS purchase_amount, COUNT(DISTINCT session_id) AS session_count FROM source_data WHERE event_timestamp \u0026gt;= \u0026#39;2022-01-01\u0026#39; GROUP BY year, month, user_id \u0026#34;\u0026#34;\u0026#34;) # Write output (partitioned if requested) timestamp = datetime.now().strftime(\u0026#34;%Y%m%d_%H%M%S\u0026#34;) if partition_by: output_path = os.path.join(output_dir, f\u0026#34;data_{timestamp}\u0026#34;) conn.execute(f\u0026#34;\u0026#34;\u0026#34; COPY transformed_data TO \u0026#39;{output_path}\u0026#39; ( FORMAT PARQUET, PARTITION_BY ({partition_by}) ) \u0026#34;\u0026#34;\u0026#34;) else: output_file = os.path.join(output_dir, f\u0026#34;data_{timestamp}.parquet\u0026#34;) conn.execute(f\u0026#34;\u0026#34;\u0026#34; COPY transformed_data TO \u0026#39;{output_file}\u0026#39; (FORMAT PARQUET) \u0026#34;\u0026#34;\u0026#34;) return conn.execute(\u0026#34;SELECT COUNT(*) FROM transformed_data\u0026#34;).fetchone()[0] if __name__ == \u0026#34;__main__\u0026#34;: parser = argparse.ArgumentParser(description=\u0026#39;Process large datasets with DuckDB\u0026#39;) parser.add_argument(\u0026#39;--input\u0026#39;, nargs=\u0026#39;+\u0026#39;, required=True, help=\u0026#39;Input file paths\u0026#39;) parser.add_argument(\u0026#39;--output-dir\u0026#39;, required=True, help=\u0026#39;Output directory\u0026#39;) parser.add_argument(\u0026#39;--partition-by\u0026#39;, help=\u0026#39;Column(s) to partition by, comma-separated\u0026#39;) args = parser.parse_args() partition_cols = args.partition_by.split(\u0026#39;,\u0026#39;) if args.partition_by else None record_count = process_large_dataset(args.input, args.output_dir, partition_cols) print(f\u0026#34;Processed {record_count} records successfully\u0026#34;) Performance Tips for DuckDB # To get the most out of DuckDB in your ETL processes, consider these optimization techniques:\nUse Appropriate Data Types: DuckDB works best when data types are appropriate for the content (integers for IDs, etc.)\nLeverage Parallelism: DuckDB can use multiple CPU cores. Enable this with:\nconn.execute(\u0026#34;PRAGMA threads=4\u0026#34;) # Adjust number based on your CPU Process Files Directly: When possible, process files directly rather than loading into memory first:\n# Instead of: df = pd.read_csv(\u0026#39;large_file.csv\u0026#39;) result = conn.execute(\u0026#34;SELECT * FROM df\u0026#34;).fetchdf() # Do this: result = conn.execute(\u0026#34;SELECT * FROM read_csv_auto(\u0026#39;large_file.csv\u0026#39;)\u0026#34;).fetchdf() Use Persistent Storage Wisely: For very large datasets, consider using a file-based database instead of in-memory:\nconn = duckdb.connect(\u0026#39;my_database.db\u0026#39;) # This will persist to disk Partition Outputs: For large output datasets, use partitioning to create more manageable files:\nconn.execute(\u0026#34;\u0026#34;\u0026#34; COPY results TO \u0026#39;output_directory\u0026#39; ( FORMAT PARQUET, PARTITION_BY (year, month) ) \u0026#34;\u0026#34;\u0026#34;) Comparing DuckDB with Alternatives # When should you choose DuckDB over alternatives like Pandas, SQL databases, or Spark?\nTool Best For Limitations DuckDB Single-node analytics, medium to large datasets, SQL users Limited transaction support, single machine only Pandas Small to medium datasets, complex transformations Memory-constrained, less SQL support Traditional SQL DB ACID transactions, complex relations Setup overhead, less analytical performance Spark Truly big data, distributed processing Complexity, overhead for smaller datasets Conclusion # DuckDB offers a powerful and easy-to-use solution for ETL processes in both Jupyter Notebooks and Python scripts. Its SQL interface, excellent performance, and seamless integration with Python data tools make it an ideal choice for many data engineering tasks.\nWhether you\u0026rsquo;re exploring data interactively in a notebook or building production-ready ETL pipelines, DuckDB can significantly simplify your workflow while providing the performance needed for modern data processing challenges.\nFor more information and advanced usage, check out the official DuckDB documentation.\nHave you tried DuckDB in your data projects? What has your experience been? Feel free to share in the comments below!\n","date":"March 11, 2025","externalUrl":null,"permalink":"/post/duckdb_jupyter_python_etl/","section":"Posts","summary":"Learn how to leverage DuckDB for efficient data processing in both Jupyter Notebooks and Python scripts with practical examples","title":"Using DuckDB for ETL: A Guide for Jupyter Notebook and Python Scripts","type":"post"},{"content":"","date":"March 11, 2025","externalUrl":null,"permalink":"/tags/web-development/","section":"Tags","summary":"","title":"Web Development","type":"tags"},{"content":"","date":"March 11, 2025","externalUrl":null,"permalink":"/tags/website-creation/","section":"Tags","summary":"","title":"Website Creation","type":"tags"},{"content":"","date":"March 11, 2024","externalUrl":null,"permalink":"/tags/ai/","section":"Tags","summary":"","title":"AI","type":"tags"},{"content":"","date":"March 11, 2024","externalUrl":null,"permalink":"/tags/computer-vision/","section":"Tags","summary":"","title":"Computer Vision","type":"tags"},{"content":" Introduction to Diffusion Self-Distillation # The GitHub repository diffusion-self-distillation by primecai presents a groundbreaking approach in the field of image generation, specifically focusing on zero-shot customized image generation. This project is an official implementation of the paper \u0026ldquo;Diffusion Self-Distillation for Zero-Shot Customized Image Generation\u0026rdquo; intended for presentation at CVPR 2025. The repository, though currently under construction, promises significant advancements in subject-preserving generation and relighting models.\nOverview of Key Features # At its core, this project leverages a pre-trained text-to-image diffusion model to generate its own dataset for text-conditioned image-to-image tasks, thereby overcoming the limitation of insufficient high-quality paired data. The method, termed Diffusion Self-Distillation, involves several key steps:\nUtilizing a text-to-image diffusion model\u0026rsquo;s capability to create grids of images. Curating a large paired dataset with the assistance of a Visual-Language Model. Fine-tuning the text-to-image model into a text+image-to-image model using the curated dataset. This approach demonstrates impressive results, outperforming existing zero-shot methods and competing with per-instance tuning techniques on various identity-preservation generation tasks without requiring test-time optimization.\nTechnical Aspects and Implementation Details # The project is implemented in Python, necessitating a significant amount of GPU memory (\u0026gt;24GB). However, the developers are working on a quantized version to support devices with less than 24GB GPU memory. The codebase includes:\nSetup and Dependencies: Cloning the repository and installing dependencies via pip install -r requirements.txt. Pretrained Models: Downloading pretrained models from Hugging Face or Google Drive, which include necessary files like config.json, diffusion_pytorch_model.safetensors, and pytorch_lora_weights.safetensors. Inference: Generating subject-preserving images using the generate.py script, specifying paths to the model, Lora weights, and condition images. Implementation Challenges # The implementation of Diffusion Self-Distillation faces several challenges, including:\nComputational Resources: The high demand for GPU memory poses a significant challenge, especially for users with limited computational resources. Model Complexity: Fine-tuning a pre-trained text-to-image model to adapt to new tasks while preserving its original capabilities is a complex task. Data Curation: Creating a large, high-quality paired dataset through the interaction of diffusion models and visual-language models requires careful curation and validation. Potential Use Cases and Applications # The capabilities of Diffusion Self-Distillation have far-reaching implications for various applications:\nArtistic Control: Providing artists with fine-grained control over text-to-image generation, enabling them to produce images of specific instances in novel contexts. Identity-Preserving Generation: Useful for tasks like relighting, where the goal is to modify an image while preserving its essential subject or identity. Customized Image Generation: Enabling zero-shot customized image generation without the need for extensive paired datasets or test-time optimization. Getting Started with Diffusion Self-Distillation # To embark on this project, follow these steps:\nClone the Repository: Start by cloning the diffusion-self-distillation repository from GitHub. Install Dependencies: Run pip install -r requirements.txt to set up your environment. Download Pretrained Models: Obtain the necessary models from Hugging Face or Google Drive. Configure Paths: Ensure you have the correct paths for the model, Lora weights, and any condition images you wish to use. Run Inference Script: Execute the generate.py script with appropriate arguments to generate subject-preserving images. For example: CUDA_VISIBLE_DEVICES=0 python generate.py \\ --model_path /PATH/TO/transformer \\ --lora_path /PATH/TO/pytorch_lora_weights.safetensors \\ --image_path /PATH/TO/conditioning_image.png \\ --text \u0026#34;this character sitting on a chair\u0026#34; \\ --output_path output.png \\ --guidance 3.5 \\ --i_guidance 1.0 \\ --t_guidance 1.0 Additional Resources # For those interested in exploring this project further, several resources are available:\nProject Website Research Paper on arXiv Hugging Face Demo Hugging Face Model Dataset on Hugging Face Conclusion # The diffusion-self-distillation repository presents a significant leap forward in the domain of image generation, particularly for tasks requiring fine-grained control and customization. By leveraging Diffusion Self-Distillation, developers and artists can achieve high-quality, identity-preserving image generation without the need for extensive datasets or complex optimization processes at test time. As this project continues to evolve, it is poised to make a substantial impact on both academic research and practical applications in computer vision and generative modeling.\nCitation # If you use this work in your research or projects, please cite the original paper:\n@inproceedings{cai2024dsd, author={Cai, Shengqu and Chan, Eric Ryan and Zhang, Yunzhi and Guibas, Leonidas and Wu, Jiajun and Wetzstein, Gordon.}, title={Diffusion Self-Distillation for Zero-Shot Customized Image Generation}, booktitle={CVPR}, year={2025} } ","date":"March 11, 2024","externalUrl":null,"permalink":"/post/exploring-diffusion-self-distillation-a-comprehensive-review/","section":"Posts","summary":"An in-depth look at the diffusion-self-distillation repository, which implements a novel approach for zero-shot customized image generation using diffusion models.","title":"Exploring diffusion-self-distillation: A Comprehensive Review","type":"post"},{"content":"","date":"March 11, 2024","externalUrl":null,"permalink":"/tags/github/","section":"Tags","summary":"","title":"GitHub","type":"tags"},{"content":"","date":"March 11, 2024","externalUrl":null,"permalink":"/tags/image-generation/","section":"Tags","summary":"","title":"Image Generation","type":"tags"},{"content":"","date":"March 11, 2024","externalUrl":null,"permalink":"/tags/open-source/","section":"Tags","summary":"","title":"Open Source","type":"tags"},{"content":" Automating Git Repository Updates and Crontab Synchronization in Unix # Introduction # Managing a Git repository and syncing Crontab files across multiple Unix machines can be tedious if done manually. Automating these tasks using shell scripts can save time and reduce the chance of errors. This article will guide you through writing a script that performs the following tasks:\nUpdate a Git Repository: Pulls the latest changes from a remote Git repository to your local machine. Grab the Crontab File: Copies the Crontab file from the repository to the local machine. Update Crontab: Applies the new Crontab settings. Prerequisites # Unix-based System: This guide assumes you\u0026rsquo;re using a Unix-based system like Linux or macOS. Git Installed: Git must be installed on your machine. Cron Service: The cron service should be enabled. Updating Git Repository # Step 1: Navigate to Git Repository # Navigate to the directory where your Git repository is located.\ncd /path/to/your/git/repo Step 2: Pull Latest Changes # Execute the git pull command to update your repository.\ngit pull origin master Grabbing the Crontab File # Step 1: Locate Crontab File in Repo # Make sure you know the path of the Crontab file within your repository.\nStep 2: Copy Crontab File to Local System # Copy the Crontab file to an appropriate location on your local system.\ncp /path/to/your/git/repo/crontab.txt /path/to/local/crontab.txt Updating Crontab # Step 1: Update Crontab from File # You can update the Crontab by reading from the file you just copied.\ncrontab /path/to/local/crontab.txt Putting It All Together: The Script # Here\u0026rsquo;s a shell script that combines all the steps.\n#!/bin/bash # Navigate to Git repository cd /path/to/your/git/repo # Update Git repository git pull origin master # Copy Crontab file to local system cp /path/to/your/git/repo/crontab.txt /path/to/local/crontab.txt # Update Crontab crontab /path/to/local/crontab.txt Step 1: Make the Script Executable # chmod +x update_git_and_crontab.sh Step 2: Run the Script # ./update_git_and_crontab.sh Comments # Make sure to replace the placeholders with the actual paths relevant to your setup. You can add this script to your own Crontab to run it automatically at intervals. Output # The Git repository will be updated. The Crontab file will be copied to the local system. Crontab will be updated according to the new file. ","date":"October 31, 2023","externalUrl":null,"permalink":"/post/git_crontab_sync/","section":"Posts","summary":"Learn how to automate the updating of a Git repository and synchronization of Crontab settings across Unix-based systems. Our step-by-step guide will walk you through the scripting and implementation.","title":"Automating Git Repository Updates and Crontab Synchronization in Unix","type":"post"},{"content":"","date":"October 31, 2023","externalUrl":null,"permalink":"/tags/automation/","section":"Tags","summary":"","title":"Automation","type":"tags"},{"content":"","date":"October 31, 2023","externalUrl":null,"permalink":"/tags/crontab/","section":"Tags","summary":"","title":"Crontab","type":"tags"},{"content":"","date":"October 31, 2023","externalUrl":null,"permalink":"/tags/devops/","section":"Tags","summary":"","title":"DevOps","type":"tags"},{"content":"","date":"October 31, 2023","externalUrl":null,"permalink":"/tags/git/","section":"Tags","summary":"","title":"Git","type":"tags"},{"content":"","date":"October 31, 2023","externalUrl":null,"permalink":"/tags/shell-scripting/","section":"Tags","summary":"","title":"Shell Scripting","type":"tags"},{"content":"","date":"October 31, 2023","externalUrl":null,"permalink":"/tags/unix/","section":"Tags","summary":"","title":"Unix","type":"tags"},{"content":"","date":"October 25, 2023","externalUrl":null,"permalink":"/tags/ad-blocking/","section":"Tags","summary":"","title":"Ad Blocking","type":"tags"},{"content":"","date":"October 25, 2023","externalUrl":null,"permalink":"/tags/debian/","section":"Tags","summary":"","title":"Debian","type":"tags"},{"content":"","date":"October 25, 2023","externalUrl":null,"permalink":"/tags/dns-sinkhole/","section":"Tags","summary":"","title":"DNS Sinkhole","type":"tags"},{"content":"","date":"October 25, 2023","externalUrl":null,"permalink":"/tags/network-wide-ad-blocking/","section":"Tags","summary":"","title":"Network-Wide Ad Blocking","type":"tags"},{"content":"","date":"October 25, 2023","externalUrl":null,"permalink":"/tags/pi-hole/","section":"Tags","summary":"","title":"Pi-Hole","type":"tags"},{"content":" PiHole: A Shield Against Unwanted Advertisements # Pi-hole is a network-wide ad blocker that acts as a DNS sinkhole, and its main function is to protect devices within a network from unwanted content like advertisements without necessitating any client-side software installations. By intercepting DNS requests, Pi-hole prevents unwanted content from being loaded, thus creating a cleaner browsing experience across your network. The setup is straightforward, with a versatile installer guiding you through the process that takes less than ten minutes. Moreover, Pi-hole\u0026rsquo;s effectiveness extends beyond browser-based ads to other areas such as ad-laden mobile apps and smart TVs, providing a more comprehensive shield against intrusive advertisements (Pi-hole documentation)(Pi-hole Network-wide Ad Blocking).\nSimple Implementation: # Device Preparation:\nPi-hole can be installed on various devices including Raspberry Pi, Ubuntu, Debian, or a virtual machine among others. For a common setup using a Raspberry Pi: Obtain a Raspberry Pi (any model with network support will do). Install Raspbian or any other supported operating system on your Raspberry Pi. Ensure your Raspberry Pi is connected to your network. Pi-hole Installation:\nOpen a terminal on your device. Run the following command: curl -sSL https://install.pi-hole.net | bash Follow the on-screen instructions to complete the installation. During the setup, you will be prompted to choose a static IP address for your Pi-hole, which is essential for its functioning as a DNS server. You\u0026rsquo;ll also choose the upstream DNS provider (like Google DNS, Cloudflare, etc.) and blocklists during the setup. Network Configuration:\nAfter installation, configure your router’s DNS settings to point to the Pi-hole. This way, all devices on your network will send their DNS requests to Pi-hole, which will filter out requests to known ad-serving domains. Optional Enhancements:\nPair your Pi-hole with a VPN to extend ad-blocking capabilities to your cellular devices, especially useful for limited bandwidth data plans. Maintenance:\nPi-hole has a web interface for monitoring and configuration. Access it by entering the Pi-hole’s IP address in your web browser. Regularly update Pi-hole to the latest version to ensure optimal performance and updated blocklists. Compatible Devices: # Raspberry Pi: All models with networking capabilities. PCs: Running Ubuntu or Debian. Virtual Machines: Any virtual machine capable of running Linux. Helpful Resources: # Official Pi-hole Documentation Pi-hole Network-wide Ad Blocking Official Site Pi-hole Setup Guide Pi-hole Reddit Community With the above steps and resources, you\u0026rsquo;re well on your way to setting up a Pi-hole in your network and enjoying an ad-free browsing experience.\n","date":"October 25, 2023","externalUrl":null,"permalink":"/post/pihole/","section":"Posts","summary":"Discover the simplicity and effectiveness of Pi-hole, a network-wide ad blocker that shields all your devices from unwanted advertisements. Our step-by-step guide walks you through a straightforward implementation on a Raspberry Pi or other compatible devices. Explore the list of devices you can use, and dive into a cleaner browsing experience.","title":"PiHole: A Shield Against Unwanted Advertisements","type":"post"},{"content":"","date":"October 25, 2023","externalUrl":null,"permalink":"/tags/privacy/","section":"Tags","summary":"","title":"Privacy","type":"tags"},{"content":"","date":"October 25, 2023","externalUrl":null,"permalink":"/tags/ubuntu/","section":"Tags","summary":"","title":"Ubuntu","type":"tags"},{"content":" Controlling a Sphero Mini with Python and spherov2 # If you\u0026rsquo;ve ever wanted to control a Sphero Mini robot using Python, you\u0026rsquo;re in the right place. In this article, we\u0026rsquo;ll explore how to use the spherov2 Python package to interact with a Sphero Mini. We\u0026rsquo;ll focus on a specific example script, controls.py, which demonstrates how to control the robot\u0026rsquo;s movements and LED lights.\nWhat is Sphero Mini? # Sphero Mini is a small, spherical robot that can roll around, change colors, and be programmed to perform various tasks. It\u0026rsquo;s a fantastic educational tool for learning about robotics and programming.\nWhat is spherov2? # The spherov2 package is a Python library that provides an API for interacting with Sphero robots, including the Sphero Mini. It allows you to control the robot\u0026rsquo;s movements, change its LED colors, and much more.\nGetting Started # To get started, you\u0026rsquo;ll need to install the spherov2 package. You can clone the repository and install the package using pip:\ngit clone https://github.com/yyakauleu/sphero_mini_with_spherov2.git cd sphero_mini_with_spherov2 pip install -r requirements.txt Controlling the Sphero Mini # The controls.py script provides a simple example of how to control the Sphero Mini. The script uses the keyboard library to detect keypress events and the spherov2 package to send commands to the robot.\nHere\u0026rsquo;s a snippet from the script:\nimport time from spherov2.sphero_edu import SpheroEduAPI, scanner from spherov2.types import Color import keyboard def main(): toy = scanner.find_toy() with SpheroEduAPI(toy) as droid: print(\u0026#34;Connected\u0026#34;) # ... (rest of the code) In this script, the main() function initializes the Sphero Mini and enters a loop that listens for keyboard events. Depending on the key pressed, the script sends different commands to the robot.\nConclusion # Controlling a Sphero Mini with Python is both fun and educational. The spherov2 package makes it easy to interact with the robot, and the controls.py script provides a great starting point for your own projects.\nFeel free to clone the repository and try it out for yourself!\n","date":"October 24, 2023","externalUrl":null,"permalink":"/post/sphero_mini_with_spherov2/","section":"Posts","summary":"Control your Sphero Mini robot effortlessly with Python using the spherov2 package. Our step-by-step guide and example script make it easy for anyone to get started. Learn how to move your Sphero Mini, change its LED colors, and more!","title":"Controlling a Sphero Mini with Python and spherov2","type":"post"},{"content":"","date":"October 24, 2023","externalUrl":null,"permalink":"/tags/educational-tool/","section":"Tags","summary":"","title":"Educational Tool","type":"tags"},{"content":"","date":"October 24, 2023","externalUrl":null,"permalink":"/tags/led-control/","section":"Tags","summary":"","title":"LED Control","type":"tags"},{"content":"","date":"October 24, 2023","externalUrl":null,"permalink":"/tags/movement-control/","section":"Tags","summary":"","title":"Movement Control","type":"tags"},{"content":"","date":"October 24, 2023","externalUrl":null,"permalink":"/tags/programming/","section":"Tags","summary":"","title":"Programming","type":"tags"},{"content":"","date":"October 24, 2023","externalUrl":null,"permalink":"/tags/robotics/","section":"Tags","summary":"","title":"Robotics","type":"tags"},{"content":"","date":"October 24, 2023","externalUrl":null,"permalink":"/tags/sphero-mini/","section":"Tags","summary":"","title":"Sphero Mini","type":"tags"},{"content":"","date":"October 24, 2023","externalUrl":null,"permalink":"/tags/spherov2/","section":"Tags","summary":"","title":"Spherov2","type":"tags"},{"content":" Databricks vs. Snowflake # VS Databricks is a cloud-based data and AI platform designed to accelerate the performance and functionality of ETL (Extract, Transform, Load) pipelines across various industries. It provides data management, security, and governance capabilities, making it suitable for a range of data-centric tasks. On the other hand, Snowflake is software that provides users with a data lake and warehousing environment for data processing, unification, and transformation, aiming to simplify complex data pipelines and enhance data integration with other tools for greater functionality (TechRepublic).\nCost: # Databricks:\nThe pricing starts at $0.07 per Databricks Unit (DBU) for data engineering work, and $0.20 per DBU for workflows and streaming. The primary cost component is the DBU, representing a unit of processing capability per hour. The total cost depends on the instance types and the number of instances used in the Databricks cluster (Databricks)(Granulate). After a free trial, Databricks offers a pay-as-you-go solution, with pricing based on computing usage. Alternatively, customers can opt for a committed use plan, committing to certain levels of usage to gain discounts when purchasing the software (TechRepublic). Snowflake:\nSimilar to Databricks, Snowflake offers a pay-as-you-go model that is usage-based with no long-term commitment, or through Snowflake On Demand, which allows customers to pre-purchase software capacity options and get discounts on the software\u0026rsquo;s overall cost (TechRepublic). Integration with Amazon S3: # Databricks can run on top of AWS S3, Azure Blob Storage, and Google Cloud Storage, allowing for storage solutions by integrating with these cloud storage platforms (eWeek). Comparison with Snowflake: # Primary Focus: Databricks is better suited for data science and massive workloads, while Snowflake is more oriented towards SQL-like business intelligence and smaller workloads (Portable). Storage Layer: Snowflake includes a storage layer, whereas Databricks provides storage by running on top of cloud storage solutions like AWS S3 (eWeek). Architecture: Snowflake is inspired by legacy warehouse architecture but modernized with decoupled storage and processing which can be scaled independently. On the other hand, Databricks has fully decoupled storage and processing layers, allowing data to be stored anywhere in any format or shape (Macrometa). Key Differences in Features: # Data Warehousing: Snowflake has a focus on data warehousing while Databricks does not. Real-time Data Analytics: Databricks supports real-time data analytics, but Snowflake does not. Built-in Machine Learning: Databricks has built-in machine learning capabilities, unlike Snowflake (TechRepublic). In conclusion, the choice between Databricks and Snowflake would depend on your organization\u0026rsquo;s specific needs, whether it\u0026rsquo;s data science, massive workloads, or SQL-like business intelligence. Additionally, the budget and the pricing model that suits your organization would also play a significant role in making this decision.\n","date":"October 23, 2023","externalUrl":null,"permalink":"/post/databricks_vs_snowflake/","section":"Posts","summary":"An in-depth comparison of Databricks and Snowflake, focusing on their costs, features, and integration capabilities with Amazon S3. This page provides insights into their pricing models, primary focus areas, architectural differences, and key features such as real-time analytics and built-in machine learning, helping organizations to make an informed choice between these two data management solutions.","title":"Comparing Databricks and Snowflake: Costs, Features, and Integration","type":"post"},{"content":"","date":"October 23, 2023","externalUrl":null,"permalink":"/tags/databricks/","section":"Tags","summary":"","title":"Databricks","type":"tags"},{"content":"","date":"October 23, 2023","externalUrl":null,"permalink":"/tags/databricks-vs.-snowflake/","section":"Tags","summary":"","title":"Databricks vs. Snowflake","type":"tags"},{"content":"","date":"October 23, 2023","externalUrl":null,"permalink":"/tags/snowflake/","section":"Tags","summary":"","title":"Snowflake","type":"tags"},{"content":" Converting TS Video to MP4 using FFmpeg # FFmpeg is a powerful, open-source, and widely-used multimedia framework that can handle a wide variety of audio and video formats. In this guide, we will focus on how to use FFmpeg to convert a video file in the Transport Stream (TS) format to the more common MP4 format.\nThe command we will be using for this conversion is:\nffmpeg -i input.ts -c:v libx264 -c:a copy output.mp4 When to Use This Command # You might need to convert a video file from the TS format to the MP4 format in the following scenarios:\nCompatibility: The TS format is typically used for broadcasting and streaming purposes, while the MP4 format is more widely supported across various platforms and devices. Converting the file to MP4 can help ensure compatibility with a broader range of media players and devices.\nStorage and Sharing: The MP4 format often provides better compression and smaller file sizes than the TS format, making it a better choice for storing and sharing videos.\nVideo Editing: The MP4 format is more compatible with popular video editing software, so converting a TS file to MP4 can be helpful when you need to edit your video.\nHow to Use This Command # To use this command to convert a TS video to MP4, you need to have FFmpeg installed on your computer. If you don\u0026rsquo;t have it installed, you can follow the instructions on the official FFmpeg website to download and install it for your operating system.\nOnce you have FFmpeg installed, follow these steps:\nOpen a terminal or command prompt.\nNavigate to the directory where your TS video file is located.\nRun the following command:\nffmpeg -i input.ts -c:v libx264 -c:a copy output.mp4 Replace input.ts with the name of your TS video file and output.mp4 with the desired name for the converted MP4 file.\nUnderstanding the Command # Let\u0026rsquo;s break down the command and understand what each part does:\nffmpeg: This is the command-line tool that is part of the FFmpeg framework. -i input.ts: This specifies the input file, which is the TS video you want to convert. -c:v libx264: This sets the video codec to libx264, which is a popular and efficient codec for encoding video in the H.264 format. This codec is widely supported and offers good compression. -c:a copy: This tells FFmpeg to copy the audio stream from the input file without re-encoding it. This is useful to preserve the original audio quality and save processing time. output.mp4: This specifies the name and format of the output file, which will be in MP4 format. After running the command, FFmpeg will convert the TS video to the MP4 format, and the output file will be saved in the same directory as the input file.\n","date":"April 9, 2023","externalUrl":null,"permalink":"/post/ts_video_to_mp4/","section":"Posts","summary":"Converting TS Video to MP4 using FFmpeg","title":"Converting TS Video to MP4 using FFmpeg","type":"post"},{"content":"","date":"April 9, 2023","externalUrl":null,"permalink":"/tags/ffmpeg-to-mp4/","section":"Tags","summary":"","title":"Ffmpeg to Mp4","type":"tags"},{"content":"","date":"April 9, 2023","externalUrl":null,"permalink":"/tags/unix-commands/","section":"Tags","summary":"","title":"Unix Commands","type":"tags"},{"content":"Hello — I\u0026rsquo;m Yauheni Yakauleu, a Quantitative Analyst \u0026amp; AI Engineer based in London. My work sits at the intersection of fundamental equities, alternative data alpha, and risk management — with the engineering muscle to turn research ideas into working systems rather than slideware.\nWhat I do # Fundamental equities — valuation, equity risk research, and disciplined quantitative framing of what a business is actually worth Alternative data \u0026amp; alpha — sourcing, cleaning, and evaluating non-traditional datasets, and separating signal from anecdata Risk management — forecasting, scenario analysis, and tail risk: VaR/ES, Monte Carlo, and friends Applied AI engineering — from classical ML (time-series forecasting with Prophet/SARIMA) to modern LLM-assisted pipelines, always with reproducible, tested code This blog # Quick Insights is where I work out loud: short, practical articles on quant methods, data engineering, and applied AI — each with working code and numbers you can reproduce from a fixed seed. If a tutorial can\u0026rsquo;t be verified against a closed form, I say so; if a heuristic has a failure mode, you\u0026rsquo;ll see it.\nA few things readers hit first:\nPricing options with Monte Carlo in NumPy — verified against Black-Scholes Predicting credit card sales: quarterly forecasting DuckDB for ETL in Jupyter \u0026amp; Python Background # I studied at The University of Texas at Arlington (competing as a student-athlete along the way — including Student Athlete of the Year in 2012), and have spent my career in the finance side of tech: quantitative analysis, alternative data, and AI implementation. I also run Code Insights, Touch IT LLC.\nFor the full CV, see LinkedIn. Want to discuss an article or a project? Use the form below — or find me on LinkedIn.\nEmail: Message: Send ","date":"March 24, 2023","externalUrl":null,"permalink":"/about/","section":"Quick Insights","summary":"Hello — I’m Yauheni Yakauleu, a Quantitative Analyst \u0026 AI Engineer based in London. My work sits at the intersection of fundamental equities, alternative data alpha, and risk management — with the engineering muscle to turn research ideas into working systems rather than slideware.\n","title":"About","type":"page"},{"content":"Interactive Data Tables in Python Using itables # When working with data in Python, displaying your dataframes in an interactive, user-friendly way can dramatically improve your analysis workflow. While pandas provides great functionality for data manipulation, its default display options are static and limited. This is where the itables library shines - it transforms your pandas DataFrames into fully interactive tables with sorting, filtering, and searching capabilities.\nIn this guide, we\u0026rsquo;ll explore how to use itables in Python, from basic installation to advanced customization.\nWhat is itables? # itables is a Python library that provides an easy way to display pandas DataFrames as interactive tables. It leverages the powerful DataTables JavaScript library under the hood but provides a simple Python interface that integrates seamlessly with your pandas workflow.\nKey features include:\nInteractive sorting by columns Filtering and searching across the entire dataset Pagination for large datasets Customizable styling and display options Support for Jupyter notebooks and standalone HTML output Installation # Installing itables is straightforward using pip:\npip install itables The library has a few dependencies, primarily pandas and ipywidgets (for Jupyter integration), which will be installed automatically if they\u0026rsquo;re not already present.\nFor the best experience in Jupyter notebooks, you should also enable the extension:\njupyter nbextension enable --py widgetsnbextension Basic Usage in Jupyter Notebook # Let\u0026rsquo;s start with some simple examples of how to use itables in a Jupyter Notebook environment:\nimport pandas as pd from itables import init_notebook_mode, show # Initialize itables for use in a Jupyter notebook with all tables interactive init_notebook_mode(all_interactive=True) # Create a sample DataFrame data = { \u0026#39;Name\u0026#39;: [\u0026#39;John Smith\u0026#39;, \u0026#39;Jane Doe\u0026#39;, \u0026#39;Robert Johnson\u0026#39;, \u0026#39;Lisa Wang\u0026#39;, \u0026#39;Michael Brown\u0026#39;], \u0026#39;Age\u0026#39;: [28, 34, 42, 25, 39], \u0026#39;City\u0026#39;: [\u0026#39;New York\u0026#39;, \u0026#39;San Francisco\u0026#39;, \u0026#39;Chicago\u0026#39;, \u0026#39;Boston\u0026#39;, \u0026#39;Seattle\u0026#39;], \u0026#39;Department\u0026#39;: [\u0026#39;Marketing\u0026#39;, \u0026#39;Engineering\u0026#39;, \u0026#39;Sales\u0026#39;, \u0026#39;Engineering\u0026#39;, \u0026#39;HR\u0026#39;], \u0026#39;Salary\u0026#39;: [72000, 105000, 83000, 98000, 79000] } df = pd.DataFrame(data) # With all_interactive=True, you don\u0026#39;t need to use show() # Simply output the DataFrame and it will be interactive df # This will display as an interactive table # You can also create another DataFrame and it will automatically be interactive sales_data = pd.DataFrame({ \u0026#39;Product\u0026#39;: [\u0026#39;Laptop\u0026#39;, \u0026#39;Phone\u0026#39;, \u0026#39;Tablet\u0026#39;, \u0026#39;Monitor\u0026#39;, \u0026#39;Headphones\u0026#39;], \u0026#39;Units_Sold\u0026#39;: [125, 310, 205, 98, 164], \u0026#39;Revenue\u0026#39;: [187500, 232500, 102500, 29400, 16400], \u0026#39;Quarter\u0026#39;: [\u0026#39;Q1\u0026#39;, \u0026#39;Q1\u0026#39;, \u0026#39;Q2\u0026#39;, \u0026#39;Q2\u0026#39;, \u0026#39;Q3\u0026#39;] }) # This will also display as an interactive table sales_data The init_notebook_mode(all_interactive=True) function initializes itables for use in a Jupyter environment and makes all DataFrame displays interactive by default. This means you don\u0026rsquo;t need to explicitly call the show() function - any DataFrame that gets displayed will automatically be rendered as an interactive table.\nUnderstanding the all_interactive Parameter # When you set all_interactive=True:\nAll pandas DataFrames displayed in output cells become interactive tables You don\u0026rsquo;t need to use the show() function explicitly It affects all DataFrames displayed after the initialization If you prefer to selectively make only certain DataFrames interactive, you can set all_interactive=False and use the show() function only on the DataFrames you want to display interactively:\n# Initialize with selective interactive mode init_notebook_mode(all_interactive=False) # This will display as a standard DataFrame (not interactive) df # This will display as an interactive table show(df) Additional Initialization Options # You can also customize the initialization with additional parameters:\n# Initialize with more options init_notebook_mode( all_interactive=True, connected=False, # Load libraries locally rather than from CDN options={\u0026#34;pageLength\u0026#34;: 25} # Set default options for all tables ) # Create and display a DataFrame large_df = pd.DataFrame({ \u0026#39;A\u0026#39;: range(100), \u0026#39;B\u0026#39;: [i * 2 for i in range(100)], \u0026#39;C\u0026#39;: [i ** 2 for i in range(100)] }) # Will display with 25 rows per page by default large_df Customizing Your Tables # One of the strengths of itables is its high degree of customizability. Let\u0026rsquo;s explore some common customization options:\nTable Length and Pagination # You can control how many rows are displayed at once:\n# Show 10 rows per page show(df, lengthMenu=[5, 10, 25, 50, 100]) Column Definitions and Formatting # For more fine-grained control over how your columns are displayed:\n# Customize column formatting show(df, columnDefs=[ # Format salary as currency {\u0026#34;targets\u0026#34;: 4, \u0026#34;render\u0026#34;: \u0026#34;$.fn.dataTable.render.number(\u0026#39;,\u0026#39;, \u0026#39;.\u0026#39;, 0, \u0026#39;$\u0026#39;)\u0026#34;}, # Make the Name column bold {\u0026#34;targets\u0026#34;: 0, \u0026#34;className\u0026#34;: \u0026#34;dt-body-bold\u0026#34;} ]) Configuring Buttons and Exports # itables supports adding buttons for exporting data:\n# Add export buttons show(df, dom=\u0026#39;Bfrtip\u0026#39;, buttons=[\u0026#39;copy\u0026#39;, \u0026#39;csv\u0026#39;, \u0026#39;excel\u0026#39;, \u0026#39;pdf\u0026#39;, \u0026#39;print\u0026#39;]) Note: To use the export buttons, you\u0026rsquo;ll need to install additional dependencies:\npip install itables[excel,pdf] Custom Styling # You can apply custom CSS to your tables:\n# Apply custom styling show(df, classes=\u0026#39;display compact stripe hover row-border\u0026#39;) Advanced Features # Now let\u0026rsquo;s explore some more advanced features of itables:\nRow Selection # Enable row selection for your tables:\n# Enable row selection show(df, select={\u0026#34;style\u0026#34;: \u0026#39;multi\u0026#39;}) Conditional Formatting # You can apply conditional formatting to highlight specific values:\n# Apply conditional formatting show(df, columnDefs=[ # Highlight salaries over 90,000 { \u0026#34;targets\u0026#34;: 4, \u0026#34;render\u0026#34;: \u0026#34;\u0026#34;\u0026#34; function(data, type, row) { if (type === \u0026#39;display\u0026#39;) { return data \u0026gt; 90000 ? \u0026#39;\u0026lt;span style=\u0026#34;color: green; font-weight: bold;\u0026#34;\u0026gt;$\u0026#39; + data + \u0026#39;\u0026lt;/span\u0026gt;\u0026#39; : \u0026#39;$\u0026#39; + data; } return data; } \u0026#34;\u0026#34;\u0026#34; } ]) Fixed Headers and Columns # For larger tables, you might want to fix headers or columns:\n# Fix headers and columns show(df, fixedHeader=True, fixedColumns={\u0026#34;leftColumns\u0026#34;: 1} ) Custom JavaScript Callbacks # Advanced users can implement custom JavaScript callbacks:\n# Add custom JavaScript callback show(df, initComplete=\u0026#34;\u0026#34;\u0026#34; function(settings, json) { console.log(\u0026#39;Table initialization complete\u0026#39;); $(this).DataTable().on(\u0026#39;select\u0026#39;, function(e, dt, type, indexes) { var data = dt.rows(indexes).data().toArray(); console.log(\u0026#39;Selected data:\u0026#39;, data); }); } \u0026#34;\u0026#34;\u0026#34; ) Using itables Outside of Jupyter # While itables works seamlessly in Jupyter notebooks, you can also use it to generate standalone HTML files:\nfrom itables.export import to_html_file # Save interactive table as HTML file to_html_file(df, \u0026#39;interactive_table.html\u0026#39;, title=\u0026#39;Employee Data\u0026#39;, caption=\u0026#39;Employee Information by Department\u0026#39;, column_filters=True) This creates a self-contained HTML file that you can open in any web browser, share with colleagues, or embed in other web applications.\nPerformance Considerations # For very large datasets, rendering interactive tables can become slow. Here are some strategies to improve performance:\nLimit the number of rows: Filter your DataFrame to include only the most relevant data. Disable complex features: Features like column filters can slow down rendering for large tables. Use server-side processing: For extremely large datasets, consider using server-side processing mode: # Enable server-side processing show(df, serverSide=True) Troubleshooting Common Issues # Here are solutions to some common issues you might encounter:\nTable Not Rendering # If your table isn\u0026rsquo;t rendering properly:\nEnsure you\u0026rsquo;ve called init_notebook_mode() at the beginning of your notebook. Check if you have JavaScript enabled in your browser. Restart your Jupyter kernel and try again. Missing Export Buttons # If export buttons aren\u0026rsquo;t working:\nMake sure you\u0026rsquo;ve installed the required dependencies: pip install itables[excel,pdf]. Check if the necessary JavaScript libraries are loading (check your browser\u0026rsquo;s console for errors). Conclusion # The itables library provides a powerful yet easy-to-use interface for creating interactive data tables in Python. By transforming static pandas DataFrames into dynamic, interactive displays, it can significantly enhance your data exploration and presentation workflows.\nWhether you\u0026rsquo;re working in a Jupyter notebook for data analysis or generating standalone HTML reports, itables offers the flexibility and customization options to meet your needs.\nHave you used itables or similar libraries in your data projects? Share your experiences in the comments below!\nAdditional Resources # itables GitHub Repository DataTables Documentation Pandas Documentation ","date":"March 11, 2023","externalUrl":null,"permalink":"/post/python-itables-interactive-tables/","section":"Posts","summary":"Interactive Data Tables in Python Using itables # When working with data in Python, displaying your dataframes in an interactive, user-friendly way can dramatically improve your analysis workflow. While pandas provides great functionality for data manipulation, its default display options are static and limited. This is where the itables library shines - it transforms your pandas DataFrames into fully interactive tables with sorting, filtering, and searching capabilities.\nIn this guide, we’ll explore how to use itables in Python, from basic installation to advanced customization.\n","title":"Interactive Data Tables in Python Using itables","type":"post"},{"content":"","date":"March 11, 2023","externalUrl":null,"permalink":"/tags/pandas/","section":"Tags","summary":"","title":"Pandas","type":"tags"},{"content":"","date":"March 11, 2023","externalUrl":null,"permalink":"/tags/visualization/","section":"Tags","summary":"","title":"Visualization","type":"tags"},{"content":"","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"}]