{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Session 3.1: Binance API with CCXT\n", "## Data Processing & Technical Analysis Foundation\n", "\n", "### Learning Objectives:\n", "- Set up and configure CCXT library for Binance\n", "- Connect to Binance with API authentication\n", "- Handle API rate limits and errors\n", "- Implement reconnection logic and fail-safes\n", "- Fetch and process real-time market data\n", "\n", "### Prerequisites:\n", "- Python 3.7+\n", "- pandas, numpy\n", "- Basic understanding of APIs\n", "- Binance API keys (for authenticated endpoints)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Setup and Environment Configuration" ] }, { "cell_type": "code", "execution_count": 95, "metadata": {}, "outputs": [], "source": [ "# Import necessary libraries\n", "import ccxt\n", "import pandas as pd\n", "import numpy as np\n", "import time\n", "import logging\n", "import os\n", "from datetime import datetime, timedelta\n", "from dotenv import load_dotenv\n", "import warnings\n", "warnings.filterwarnings('ignore')\n", "\n", "# Configure logging\n", "logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Secure API Key Management with dotenv\n", "\n", "### Why Use Environment Variables?\n", "- **Security**: API keys should never be hardcoded in source code\n", "- **Flexibility**: Easy to switch between different API keys\n", "- **Version Control Safety**: .env files are excluded from Git repositories\n", "- **Team Collaboration**: Each developer can have their own credentials\n", "\n", "### Setting Up dotenv\n", "The `python-dotenv` library allows us to load environment variables from a `.env` file.\n", "\n", "**Installation**: `pip install python-dotenv`" ] }, { "cell_type": "code", "execution_count": 96, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "šŸ” Environment Variables Setup\n", "==============================\n", "šŸ“„ .env file exists: āŒ No\n", "\n", "šŸ“ Create a .env file with the following structure:\n", "\n", "# .env file template\n", "# Copy this to a new file named '.env' in your project directory\n", "\n", "# Binance API Keys (get from https://www.binance.com/en/my/settings/api-management)\n", "BINANCE_API_KEY=your_binance_api_key_here\n", "BINANCE_SECRET_KEY=your_binance_secret_key_here\n", "\n", "# General Settings\n", "ENABLE_RATE_LIMIT=true\n", "REQUEST_TIMEOUT=30000\n", " \n", "\n", "āš ļø IMPORTANT SECURITY NOTES:\n", "1. Add '.env' to your .gitignore file\n", "2. Never commit API keys to version control\n", "3. Use read-only permissions for learning/analysis\n", "4. Regularly rotate your API keys\n", "5. Limit API key permissions to minimum required\n" ] } ], "source": [ "# Load environment variables from .env file\n", "load_dotenv()\n", "\n", "# Verify dotenv is working\n", "print(\"šŸ” Environment Variables Setup\")\n", "print(\"=\" * 30)\n", "\n", "# Check if .env file exists\n", "env_file_exists = os.path.exists('.env')\n", "print(f\"šŸ“„ .env file exists: {'āœ… Yes' if env_file_exists else 'āŒ No'}\")\n", "\n", "if not env_file_exists:\n", " print(\"\\nšŸ“ Create a .env file with the following structure:\")\n", " print(\"\"\"\n", "# .env file template\n", "# Copy this to a new file named '.env' in your project directory\n", "\n", "# Binance API Keys (get from https://www.binance.com/en/my/settings/api-management)\n", "BINANCE_API_KEY=your_binance_api_key_here\n", "BINANCE_SECRET_KEY=your_binance_secret_key_here\n", "\n", "# General Settings\n", "ENABLE_RATE_LIMIT=true\n", "REQUEST_TIMEOUT=30000\n", " \"\"\")\n", "else:\n", " print(\"\\nšŸ” Checking available environment variables:\")\n", " # Check for Binance API key variables (without exposing values)\n", " env_vars_to_check = [\n", " 'BINANCE_API_KEY', 'BINANCE_SECRET_KEY', 'ENABLE_RATE_LIMIT'\n", " ]\n", " \n", " for var in env_vars_to_check:\n", " value = os.getenv(var)\n", " if value:\n", " # Show first 4 and last 4 characters for API keys, or full value for settings\n", " if 'API_KEY' in var or 'SECRET' in var:\n", " display_value = f\"{value[:4]}...{value[-4:]}\" if len(value) > 8 else \"***\"\n", " else:\n", " display_value = value\n", " print(f\" āœ… {var}: {display_value}\")\n", " else:\n", " print(f\" āŒ {var}: Not set\")\n", "\n", "print(\"\\nāš ļø IMPORTANT SECURITY NOTES:\")\n", "print(\"1. Add '.env' to your .gitignore file\")\n", "print(\"2. Never commit API keys to version control\")\n", "print(\"3. Use read-only permissions for learning/analysis\")\n", "print(\"4. Regularly rotate your API keys\")\n", "print(\"5. Limit API key permissions to minimum required\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Creating Your .env File\n", "\n", "**Step 1**: Create a file named `.env` in your project directory\n", "\n", "**Step 2**: Add your Binance API credentials:\n", "\n", "```bash\n", "# Binance Configuration\n", "BINANCE_API_KEY=your_actual_binance_api_key_here\n", "BINANCE_SECRET_KEY=your_actual_binance_secret_key_here\n", "\n", "# Settings\n", "ENABLE_RATE_LIMIT=true\n", "REQUEST_TIMEOUT=30000\n", "```\n", "\n", "### Getting Binance API Keys\n", "\n", "1. **Login** to your Binance account\n", "2. **Navigate** to API Management: Account ā†’ API Management\n", "3. **Create** a new API key with a descriptive label\n", "4. **Set Permissions**: For learning, enable only:\n", " - āœ… Read Info (spot & margin)\n", " - āŒ Spot & Margin Trading (disable for safety)\n", " - āŒ Futures (disable for safety)\n", " - āŒ Withdrawals (never enable for learning)\n", "5. **IP Restrictions**: Add your IP address for extra security\n", "6. **Copy** both the API Key and Secret Key to your .env file\n", "\n", "### API Key Security Best Practices\n", "\n", "1. **Minimal Permissions**: Only grant read permissions for learning\n", "2. **IP Restrictions**: Restrict API access to your specific IP address\n", "3. **Key Rotation**: Regularly rotate your API keys (monthly recommended)\n", "4. **Monitor Usage**: Check API usage logs regularly for unusual activity\n", "5. **Never Share**: Never share API keys in chat, email, or code repositories" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. CCXT Library Overview" ] }, { "cell_type": "code", "execution_count": 97, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CCXT Version: 4.4.85\n", "Total Supported Exchanges: 105\n", "\n", "Binance Support: āœ… Yes\n", "\n", "Other Popular Exchanges Supported by CCXT:\n", " āœ“ coinbase\n", " āœ“ kraken\n", " āœ“ bitfinex\n", " āœ“ huobi\n", " āœ“ okx\n", " āœ“ kucoin\n", " āœ“ bybit\n" ] } ], "source": [ "# Check CCXT version and Binance support\n", "print(f\"CCXT Version: {ccxt.__version__}\")\n", "print(f\"Total Supported Exchanges: {len(ccxt.exchanges)}\")\n", "print(f\"\\nBinance Support: {'āœ… Yes' if 'binance' in ccxt.exchanges else 'āŒ No'}\")\n", "\n", "# Show some popular exchanges that CCXT supports\n", "print(\"\\nOther Popular Exchanges Supported by CCXT:\")\n", "popular_exchanges = ['coinbase', 'kraken', 'bitfinex', 'huobi', 'okx', 'kucoin', 'bybit']\n", "for exchange in popular_exchanges:\n", " if exchange in ccxt.exchanges:\n", " print(f\" āœ“ {exchange}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Connecting to Binance with API Authentication" ] }, { "cell_type": "code", "execution_count": 98, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2025-06-18 20:09:06,681 - WARNING - No Binance credentials found, using public API only\n", "2025-06-18 20:09:09,288 - INFO - Successfully connected to Binance\n", "2025-06-18 20:09:09,288 - INFO - Available markets: 3730\n" ] } ], "source": [ "def get_binance_credentials():\n", " \"\"\"\n", " Retrieve Binance API credentials from environment variables\n", " \"\"\"\n", " api_key = os.getenv('BINANCE_API_KEY')\n", " secret = os.getenv('BINANCE_SECRET_KEY')\n", " \n", " if api_key and secret:\n", " return {\n", " 'apiKey': api_key,\n", " 'secret': secret,\n", " }\n", " return {}\n", "\n", "def initialize_binance(use_credentials=True):\n", " \"\"\"\n", " Initialize Binance connection with optional API authentication\n", " \"\"\"\n", " try:\n", " # Base configuration\n", " config = {\n", " 'enableRateLimit': True, # Enable built-in rate limiting\n", " 'timeout': int(os.getenv('REQUEST_TIMEOUT', 30000)), # 30 seconds timeout\n", " }\n", " \n", " # Add API credentials if available and requested\n", " if use_credentials:\n", " credentials = get_binance_credentials()\n", " if credentials:\n", " config.update(credentials)\n", " logging.info(\"Using authenticated connection for Binance\")\n", " else:\n", " logging.warning(\"No Binance credentials found, using public API only\")\n", " \n", " # Initialize Binance exchange\n", " binance = ccxt.binance(config)\n", " \n", " # Test connection\n", " markets = binance.load_markets()\n", " logging.info(f\"Successfully connected to Binance\")\n", " logging.info(f\"Available markets: {len(markets)}\")\n", " \n", " # Test authentication if credentials were provided\n", " if use_credentials and config.get('apiKey'):\n", " try:\n", " # Try to fetch account info (requires authentication)\n", " account = binance.fetch_balance()\n", " logging.info(\"āœ… Authentication successful for Binance\")\n", " except ccxt.AuthenticationError:\n", " logging.error(\"āŒ Authentication failed for Binance - check your API keys\")\n", " except ccxt.PermissionDenied:\n", " logging.warning(\"āš ļø Limited permissions for Binance - some features may not work\")\n", " except Exception as e:\n", " logging.warning(f\"Could not test authentication for Binance: {str(e)}\")\n", " \n", " return binance\n", " \n", " except Exception as e:\n", " logging.error(f\"Failed to connect to Binance: {str(e)}\")\n", " return None\n", "\n", "# Initialize Binance connection\n", "binance = initialize_binance(use_credentials=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Market Data Exploration" ] }, { "cell_type": "code", "execution_count": 99, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "BTC Trading pairs (first 10): ['BTC/USDT', 'BTC/USDT:USDT', 'BTC/USDT:USDT-250627', 'BTC/USDT:USDT-250926', 'BTCDOM/USDT:USDT', 'BTCDOWN/USDT', 'BTCST/USDT', 'BTCST/USDT:USDT', 'BTCUP/USDT', 'PUMPBTC/USDT:USDT']\n", "\n", "Available popular pairs: ['BTC/USDT', 'ETH/USDT', 'BNB/USDT', 'ADA/USDT', 'SOL/USDT', 'DOT/USDT']\n", "\n", "BTC/USDT Market Info:\n", " Base: BTC\n", " Quote: USDT\n", " Active: True\n", " Spot: True\n", " Maker Fee: 0.001\n", " Taker Fee: 0.001\n" ] } ], "source": [ "# Explore available trading pairs on Binance\n", "if binance:\n", " # Get BTC trading pairs\n", " btc_pairs = [symbol for symbol in binance.symbols if 'BTC' in symbol and '/USDT' in symbol]\n", " print(f\"BTC Trading pairs (first 10): {btc_pairs[:10]}\")\n", " \n", " # Get popular trading pairs\n", " popular_pairs = ['BTC/USDT', 'ETH/USDT', 'BNB/USDT', 'ADA/USDT', 'SOL/USDT', 'DOT/USDT']\n", " available_pairs = [pair for pair in popular_pairs if pair in binance.symbols]\n", " print(f\"\\nAvailable popular pairs: {available_pairs}\")\n", " \n", " # Show market info for BTC/USDT\n", " btc_market = binance.market('BTC/USDT')\n", " print(f\"\\nBTC/USDT Market Info:\")\n", " print(f\" Base: {btc_market['base']}\")\n", " print(f\" Quote: {btc_market['quote']}\")\n", " print(f\" Active: {btc_market['active']}\")\n", " print(f\" Spot: {btc_market['spot']}\")\n", " print(f\" Maker Fee: {btc_market['maker']}\")\n", " print(f\" Taker Fee: {btc_market['taker']}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6. Basic Data Fetching" ] }, { "cell_type": "code", "execution_count": 100, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Current Market Prices:\n", "========================================\n", "BTC/USDT: $104,165.23 (+0.28%)\n", " Volume: 13,936.07 BTC\n", " Bid/Ask: $104,165.22 / $104,165.23\n", "\n", "ETH/USDT: $2,488.32 (+0.69%)\n", " Volume: 523,024.45 ETH\n", " Bid/Ask: $2,488.31 / $2,488.32\n", "\n", "BNB/USDT: $640.48 (-0.36%)\n", " Volume: 141,740.24 BNB\n", " Bid/Ask: $640.48 / $640.49\n", "\n", "ADA/USDT: $0.59 (-2.16%)\n", " Volume: 101,214,201.90 ADA\n", " Bid/Ask: $0.59 / $0.59\n", "\n" ] } ], "source": [ "def fetch_ticker(symbol):\n", " \"\"\"\n", " Fetch current ticker data with error handling\n", " \"\"\"\n", " try:\n", " ticker = binance.fetch_ticker(symbol)\n", " return {\n", " 'symbol': symbol,\n", " 'last': ticker['last'],\n", " 'bid': ticker['bid'],\n", " 'ask': ticker['ask'],\n", " 'volume': ticker['baseVolume'],\n", " 'change': ticker['change'],\n", " 'percentage': ticker['percentage'],\n", " 'timestamp': datetime.fromtimestamp(ticker['timestamp'] / 1000)\n", " }\n", " except Exception as e:\n", " logging.error(f\"Error fetching ticker for {symbol}: {str(e)}\")\n", " return None\n", "\n", "# Fetch ticker data for popular pairs\n", "if binance:\n", " print(\"Current Market Prices:\")\n", " print(\"=\" * 40)\n", " \n", " for symbol in ['BTC/USDT', 'ETH/USDT', 'BNB/USDT', 'ADA/USDT']:\n", " ticker_data = fetch_ticker(symbol)\n", " if ticker_data:\n", " print(f\"{symbol}: ${ticker_data['last']:,.2f} ({ticker_data['percentage']:+.2f}%)\")\n", " print(f\" Volume: {ticker_data['volume']:,.2f} {symbol.split('/')[0]}\")\n", " print(f\" Bid/Ask: ${ticker_data['bid']:,.2f} / ${ticker_data['ask']:,.2f}\")\n", " print()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7. OHLCV Data Fetching" ] }, { "cell_type": "code", "execution_count": 101, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2025-06-18 20:09:10,790 - INFO - Fetched 100 candles for BTC/USDT (1h)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "BTC/USDT OHLCV Data (last 5 candles):\n", " open high low close volume\n", "datetime \n", "2025-06-18 14:00:00 104439.82 105325.83 104228.30 104619.88 1418.42072\n", "2025-06-18 15:00:00 104619.87 104973.79 104150.13 104801.01 869.24120\n", "2025-06-18 16:00:00 104801.01 104836.64 104334.78 104426.93 509.29545\n", "2025-06-18 17:00:00 104426.93 104508.00 103615.33 104331.83 918.22252\n", "2025-06-18 18:00:00 104332.00 104706.08 104011.42 104165.23 265.75381\n", "\n", "Data shape: (100, 5)\n", "Date range: 2025-06-14 15:00:00 to 2025-06-18 18:00:00\n", "\n", "Price Statistics:\n", " Highest: $108,952.38\n", " Lowest: $103,371.02\n", " Average Volume: 539.80 BTC\n" ] } ], "source": [ "def fetch_ohlcv_data(symbol, timeframe='1h', limit=100):\n", " \"\"\"\n", " Fetch OHLCV data with error handling and data processing\n", " \"\"\"\n", " try:\n", " # Fetch raw OHLCV data\n", " ohlcv = binance.fetch_ohlcv(symbol, timeframe, limit=limit)\n", " \n", " # Convert to DataFrame\n", " df = pd.DataFrame(ohlcv, columns=['timestamp', 'open', 'high', 'low', 'close', 'volume'])\n", " df['datetime'] = pd.to_datetime(df['timestamp'], unit='ms')\n", " df.set_index('datetime', inplace=True)\n", " df.drop('timestamp', axis=1, inplace=True)\n", " \n", " # Ensure numeric types\n", " for col in ['open', 'high', 'low', 'close', 'volume']:\n", " df[col] = pd.to_numeric(df[col], errors='coerce')\n", " \n", " logging.info(f\"Fetched {len(df)} candles for {symbol} ({timeframe})\")\n", " return df\n", " \n", " except Exception as e:\n", " logging.error(f\"Error fetching OHLCV data for {symbol}: {str(e)}\")\n", " return None\n", "\n", "# Fetch OHLCV data example\n", "if binance:\n", " btc_data = fetch_ohlcv_data('BTC/USDT', '1h', 100)\n", " if btc_data is not None:\n", " print(\"BTC/USDT OHLCV Data (last 5 candles):\")\n", " print(btc_data.tail())\n", " print(f\"\\nData shape: {btc_data.shape}\")\n", " print(f\"Date range: {btc_data.index.min()} to {btc_data.index.max()}\")\n", " \n", " # Show basic statistics\n", " print(f\"\\nPrice Statistics:\")\n", " print(f\" Highest: ${btc_data['high'].max():,.2f}\")\n", " print(f\" Lowest: ${btc_data['low'].min():,.2f}\")\n", " print(f\" Average Volume: {btc_data['volume'].mean():,.2f} BTC\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8. Rate Limiting and Error Management" ] }, { "cell_type": "code", "execution_count": 102, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Testing robust data fetching...\n", "Fetching BTC/USDT...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2025-06-18 20:09:11,174 - INFO - Fetched 50 candles for BTC/USDT (1h)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "āœ“ BTC/USDT: 50 candles\n", "Fetching ETH/USDT...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2025-06-18 20:09:12,484 - INFO - Fetched 50 candles for ETH/USDT (1h)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "āœ“ ETH/USDT: 50 candles\n", "Fetching BNB/USDT...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2025-06-18 20:09:13,869 - INFO - Fetched 50 candles for BNB/USDT (1h)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "āœ“ BNB/USDT: 50 candles\n", "Fetching ADA/USDT...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2025-06-18 20:09:15,178 - INFO - Fetched 50 candles for ADA/USDT (1h)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "āœ“ ADA/USDT: 50 candles\n", "\n", "Total time: 4.38 seconds\n", "Successfully fetched data for 4 symbols\n" ] } ], "source": [ "import time\n", "from functools import wraps\n", "\n", "def rate_limit_decorator(calls_per_second=1):\n", " \"\"\"\n", " Decorator to implement rate limiting\n", " \"\"\"\n", " min_interval = 1.0 / calls_per_second\n", " last_called = [0.0]\n", " \n", " def decorator(func):\n", " @wraps(func)\n", " def wrapper(*args, **kwargs):\n", " elapsed = time.time() - last_called[0]\n", " left_to_wait = min_interval - elapsed\n", " if left_to_wait > 0:\n", " time.sleep(left_to_wait)\n", " ret = func(*args, **kwargs)\n", " last_called[0] = time.time()\n", " return ret\n", " return wrapper\n", " return decorator\n", "\n", "def exponential_backoff(max_retries=3, base_delay=1):\n", " \"\"\"\n", " Decorator for exponential backoff retry logic\n", " \"\"\"\n", " def decorator(func):\n", " @wraps(func)\n", " def wrapper(*args, **kwargs):\n", " for attempt in range(max_retries + 1):\n", " try:\n", " return func(*args, **kwargs)\n", " except ccxt.NetworkError as e:\n", " if attempt == max_retries:\n", " logging.error(f\"Max retries reached for {func.__name__}: {str(e)}\")\n", " raise\n", " delay = base_delay * (2 ** attempt)\n", " logging.warning(f\"Network error in {func.__name__}, retrying in {delay}s... (attempt {attempt + 1}/{max_retries + 1})\")\n", " time.sleep(delay)\n", " except ccxt.RateLimitExceeded as e:\n", " if attempt == max_retries:\n", " logging.error(f\"Rate limit exceeded for {func.__name__}: {str(e)}\")\n", " raise\n", " delay = base_delay * (4 ** attempt) # Longer delay for rate limits\n", " logging.warning(f\"Rate limit exceeded in {func.__name__}, waiting {delay}s... (attempt {attempt + 1}/{max_retries + 1})\")\n", " time.sleep(delay)\n", " except Exception as e:\n", " logging.error(f\"Unexpected error in {func.__name__}: {str(e)}\")\n", " raise\n", " return None\n", " return wrapper\n", " return decorator\n", "\n", "# Enhanced data fetching with error handling\n", "@exponential_backoff(max_retries=3, base_delay=1)\n", "@rate_limit_decorator(calls_per_second=1)\n", "def robust_fetch_ohlcv(symbol, timeframe='1h', limit=100):\n", " \"\"\"\n", " Robust OHLCV fetching with rate limiting and error handling\n", " \"\"\"\n", " return fetch_ohlcv_data(symbol, timeframe, limit)\n", "\n", "# Test robust fetching\n", "if binance:\n", " print(\"Testing robust data fetching...\")\n", " start_time = time.time()\n", " \n", " # Fetch data for multiple symbols\n", " symbols = ['BTC/USDT', 'ETH/USDT', 'BNB/USDT', 'ADA/USDT']\n", " data = {}\n", " \n", " for symbol in symbols:\n", " print(f\"Fetching {symbol}...\")\n", " df = robust_fetch_ohlcv(symbol, '1h', 50)\n", " if df is not None:\n", " data[symbol] = df\n", " print(f\"āœ“ {symbol}: {len(df)} candles\")\n", " else:\n", " print(f\"āœ— Failed to fetch {symbol}\")\n", " \n", " end_time = time.time()\n", " print(f\"\\nTotal time: {end_time - start_time:.2f} seconds\")\n", " print(f\"Successfully fetched data for {len(data)} symbols\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9. Reconnection Logic and Connection Monitoring" ] }, { "cell_type": "code", "execution_count": 103, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Testing BinanceManager...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2025-06-18 20:09:17,482 - INFO - Connected to Binance\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "āœ“ Binance manager is healthy\n", "āœ“ BTC/USDT: $104,164.51\n", "āœ“ ETH/USDT OHLCV: 10 candles\n", "āœ“ BTC/USDT Order Book: 5 bids, 5 asks\n" ] } ], "source": [ "class BinanceManager:\n", " \"\"\"\n", " Binance manager with connection monitoring and auto-reconnection\n", " \"\"\"\n", " \n", " def __init__(self, use_credentials=True):\n", " self.use_credentials = use_credentials\n", " self.exchange = None\n", " self.last_successful_call = None\n", " self.connection_failures = 0\n", " self.max_failures = 5\n", " \n", " self._connect()\n", " \n", " def _connect(self):\n", " \"\"\"\n", " Establish connection to Binance\n", " \"\"\"\n", " try:\n", " # Base configuration\n", " config = {\n", " 'enableRateLimit': True,\n", " 'timeout': int(os.getenv('REQUEST_TIMEOUT', 30000)),\n", " }\n", " \n", " # Add credentials if requested\n", " if self.use_credentials:\n", " credentials = get_binance_credentials()\n", " if credentials:\n", " config.update(credentials)\n", " \n", " self.exchange = ccxt.binance(config)\n", " \n", " # Test connection\n", " self.exchange.load_markets()\n", " self.connection_failures = 0\n", " self.last_successful_call = datetime.now()\n", " \n", " logging.info(f\"Connected to Binance\")\n", " return True\n", " \n", " except Exception as e:\n", " logging.error(f\"Failed to connect to Binance: {str(e)}\")\n", " self.connection_failures += 1\n", " return False\n", " \n", " def _reconnect_if_needed(self):\n", " \"\"\"\n", " Reconnect if connection seems stale or failed\n", " \"\"\"\n", " if (self.connection_failures >= self.max_failures or \n", " self.last_successful_call is None or \n", " (datetime.now() - self.last_successful_call).seconds > 300): # 5 minutes\n", " \n", " logging.warning(f\"Reconnecting to Binance...\")\n", " return self._connect()\n", " return True\n", " \n", " def execute_with_retry(self, func, *args, **kwargs):\n", " \"\"\"\n", " Execute exchange function with automatic retry and reconnection\n", " \"\"\"\n", " max_retries = 3\n", " \n", " for attempt in range(max_retries):\n", " try:\n", " # Check if reconnection is needed\n", " if not self._reconnect_if_needed():\n", " continue\n", " \n", " # Execute the function\n", " result = func(*args, **kwargs)\n", " \n", " # Update success timestamp\n", " self.last_successful_call = datetime.now()\n", " self.connection_failures = 0\n", " \n", " return result\n", " \n", " except (ccxt.NetworkError, ccxt.ExchangeNotAvailable) as e:\n", " self.connection_failures += 1\n", " logging.warning(f\"Network/Exchange error (attempt {attempt + 1}): {str(e)}\")\n", " \n", " if attempt < max_retries - 1:\n", " time.sleep(2 ** attempt) # Exponential backoff\n", " \n", " except ccxt.RateLimitExceeded as e:\n", " logging.warning(f\"Rate limit exceeded (attempt {attempt + 1}): {str(e)}\")\n", " \n", " if attempt < max_retries - 1:\n", " time.sleep(5 * (attempt + 1)) # Longer wait for rate limits\n", " \n", " except Exception as e:\n", " logging.error(f\"Unexpected error: {str(e)}\")\n", " break\n", " \n", " logging.error(f\"Failed to execute function after {max_retries} attempts\")\n", " return None\n", " \n", " def fetch_ticker(self, symbol):\n", " return self.execute_with_retry(self.exchange.fetch_ticker, symbol)\n", " \n", " def fetch_ohlcv(self, symbol, timeframe='1h', limit=100):\n", " return self.execute_with_retry(self.exchange.fetch_ohlcv, symbol, timeframe, limit=limit)\n", " \n", " def fetch_balance(self):\n", " \"\"\"Fetch account balance (requires authentication)\"\"\"\n", " return self.execute_with_retry(self.exchange.fetch_balance)\n", " \n", " def fetch_order_book(self, symbol, limit=100):\n", " \"\"\"Fetch order book for a symbol\"\"\"\n", " return self.execute_with_retry(self.exchange.fetch_order_book, symbol, limit)\n", " \n", " def is_healthy(self):\n", " \"\"\"\n", " Check if connection is healthy\n", " \"\"\"\n", " return (self.connection_failures < self.max_failures and \n", " self.last_successful_call is not None and \n", " (datetime.now() - self.last_successful_call).seconds < 300)\n", "\n", "# Test the BinanceManager\n", "print(\"Testing BinanceManager...\")\n", "manager = BinanceManager(use_credentials=True)\n", "\n", "if manager.is_healthy():\n", " print(\"āœ“ Binance manager is healthy\")\n", " \n", " # Test ticker fetching\n", " ticker = manager.fetch_ticker('BTC/USDT')\n", " if ticker:\n", " print(f\"āœ“ BTC/USDT: ${ticker['last']:,.2f}\")\n", " \n", " # Test OHLCV fetching\n", " ohlcv = manager.fetch_ohlcv('ETH/USDT', '1h', 10)\n", " if ohlcv:\n", " print(f\"āœ“ ETH/USDT OHLCV: {len(ohlcv)} candles\")\n", " \n", " # Test order book fetching\n", " order_book = manager.fetch_order_book('BTC/USDT', 5)\n", " if order_book:\n", " print(f\"āœ“ BTC/USDT Order Book: {len(order_book['bids'])} bids, {len(order_book['asks'])} asks\")\n", " \n", " # Test authenticated endpoint (if credentials are available)\n", " if manager.use_credentials and manager.exchange.apiKey:\n", " try:\n", " balance = manager.fetch_balance()\n", " if balance:\n", " print(\"āœ“ Balance fetched successfully (authenticated)\")\n", " except Exception as e:\n", " print(f\"āš ļø Could not fetch balance: {str(e)}\")\n", "else:\n", " print(\"āœ— Binance manager is not healthy\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 10. Advanced Data Analysis" ] }, { "cell_type": "code", "execution_count": 104, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Market Analysis Report\n", "==================================================\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2025-06-18 20:09:19,264 - INFO - Fetched 100 candles for BTC/USDT (1h)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "BTC/USDT Analysis:\n", " Current Price: $104,149.50\n", " 24h Change: -0.25%\n", " Price Range: $103,371.02 - $108,952.38\n", " Average Volume: 539.81\n", " Volatility: 1.33%\n", " SMA 20: $104,752.94\n", " SMA 50: $105,660.09\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2025-06-18 20:09:19,585 - INFO - Fetched 100 candles for ETH/USDT (1h)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "ETH/USDT Analysis:\n", " Current Price: $2,487.00\n", " 24h Change: -0.37%\n", " Price Range: $2,453.80 - $2,680.34\n", " Average Volume: 22,377.63\n", " Volatility: 2.53%\n", " SMA 20: $2,514.66\n", " SMA 50: $2,545.11\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2025-06-18 20:09:19,872 - INFO - Fetched 100 candles for BNB/USDT (1h)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "BNB/USDT Analysis:\n", " Current Price: $640.38\n", " 24h Change: -0.88%\n", " Price Range: $637.35 - $659.46\n", " Average Volume: 5,546.83\n", " Volatility: 1.53%\n", " SMA 20: $646.92\n", " SMA 50: $650.47\n" ] } ], "source": [ "def analyze_market_data(symbol, timeframe='1h', limit=100):\n", " \"\"\"\n", " Comprehensive market data analysis\n", " \"\"\"\n", " if not binance:\n", " print(\"Binance connection not available\")\n", " return None\n", " \n", " try:\n", " # Fetch OHLCV data\n", " df = fetch_ohlcv_data(symbol, timeframe, limit)\n", " if df is None:\n", " return None\n", " \n", " # Calculate additional metrics\n", " df['price_change'] = df['close'].pct_change()\n", " df['high_low_range'] = df['high'] - df['low']\n", " df['body_size'] = abs(df['close'] - df['open'])\n", " df['upper_wick'] = df['high'] - df[['open', 'close']].max(axis=1)\n", " df['lower_wick'] = df[['open', 'close']].min(axis=1) - df['low']\n", " \n", " # Calculate moving averages\n", " df['sma_20'] = df['close'].rolling(window=20).mean()\n", " df['sma_50'] = df['close'].rolling(window=50).mean()\n", " \n", " # Calculate volatility\n", " df['volatility'] = df['price_change'].rolling(window=20).std() * np.sqrt(24) # 24-hour volatility\n", " \n", " # Analysis results\n", " analysis = {\n", " 'symbol': symbol,\n", " 'timeframe': timeframe,\n", " 'period': f\"{df.index.min()} to {df.index.max()}\",\n", " 'current_price': df['close'].iloc[-1],\n", " 'price_change_24h': df['price_change'].iloc[-24:].sum() if len(df) >= 24 else None,\n", " 'highest_price': df['high'].max(),\n", " 'lowest_price': df['low'].min(),\n", " 'average_volume': df['volume'].mean(),\n", " 'current_volatility': df['volatility'].iloc[-1] if not pd.isna(df['volatility'].iloc[-1]) else None,\n", " 'sma_20': df['sma_20'].iloc[-1] if not pd.isna(df['sma_20'].iloc[-1]) else None,\n", " 'sma_50': df['sma_50'].iloc[-1] if not pd.isna(df['sma_50'].iloc[-1]) else None,\n", " }\n", " \n", " return analysis, df\n", " \n", " except Exception as e:\n", " logging.error(f\"Error analyzing market data for {symbol}: {str(e)}\")\n", " return None\n", "\n", "# Analyze multiple symbols\n", "if binance:\n", " print(\"Market Analysis Report\")\n", " print(\"=\" * 50)\n", " \n", " symbols_to_analyze = ['BTC/USDT', 'ETH/USDT', 'BNB/USDT']\n", " \n", " for symbol in symbols_to_analyze:\n", " result = analyze_market_data(symbol, '1h', 100)\n", " if result:\n", " analysis, df = result\n", " \n", " print(f\"\\n{symbol} Analysis:\")\n", " print(f\" Current Price: ${analysis['current_price']:,.2f}\")\n", " if analysis['price_change_24h']:\n", " print(f\" 24h Change: {analysis['price_change_24h']:.2%}\")\n", " print(f\" Price Range: ${analysis['lowest_price']:,.2f} - ${analysis['highest_price']:,.2f}\")\n", " print(f\" Average Volume: {analysis['average_volume']:,.2f}\")\n", " if analysis['current_volatility']:\n", " print(f\" Volatility: {analysis['current_volatility']:.2%}\")\n", " if analysis['sma_20']:\n", " print(f\" SMA 20: ${analysis['sma_20']:,.2f}\")\n", " if analysis['sma_50']:\n", " print(f\" SMA 50: ${analysis['sma_50']:,.2f}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 11. Exercise: Build Your Data Pipeline" ] }, { "cell_type": "code", "execution_count": 105, "metadata": {}, "outputs": [], "source": [ "# Exercise: Create a data pipeline that:\n", "# 1. Fetches data from Binance for multiple symbols\n", "# 2. Handles errors gracefully\n", "# 3. Stores data in a structured format\n", "# 4. Monitors data quality\n", "# 5. Provides summary statistics\n", "\n", "class BinanceDataPipeline:\n", " def __init__(self, symbols, timeframe='1h', use_credentials=True):\n", " self.symbols = symbols\n", " self.timeframe = timeframe\n", " self.manager = BinanceManager(use_credentials=use_credentials)\n", " self.data_store = {}\n", " self.quality_report = {}\n", " \n", " def fetch_all_data(self, limit=100):\n", " \"\"\"\n", " TODO: Implement data fetching for all symbols\n", " - Fetch OHLCV data for each symbol\n", " - Store results in self.data_store\n", " - Handle errors for individual symbols\n", " \"\"\"\n", " pass\n", " \n", " def validate_data(self, symbol, df):\n", " \"\"\"\n", " TODO: Implement data quality checks\n", " - Check for missing values\n", " - Validate price ranges (no negative prices)\n", " - Check for duplicate timestamps\n", " - Validate volume data\n", " - Return quality score and issues found\n", " \"\"\"\n", " pass\n", " \n", " def generate_summary(self):\n", " \"\"\"\n", " TODO: Generate summary statistics\n", " - Calculate correlation between symbols\n", " - Show price changes\n", " - Volume analysis\n", " - Data quality summary\n", " \"\"\"\n", " pass\n", " \n", " def run_pipeline(self, limit=100):\n", " \"\"\"\n", " TODO: Implement the complete pipeline\n", " - Fetch all data\n", " - Validate each dataset\n", " - Generate summary report\n", " - Return results\n", " \"\"\"\n", " pass\n", "\n", "# Your implementation here:\n", "\n", "\n", "# Test your pipeline\n", "# pipeline = BinanceDataPipeline(\n", "# symbols=['BTC/USDT', 'ETH/USDT', 'BNB/USDT', 'ADA/USDT'],\n", "# timeframe='1h',\n", "# use_credentials=True\n", "# )\n", "# results = pipeline.run_pipeline(limit=100)\n", "# print(results)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Summary\n", "\n", "In this notebook, you learned:\n", "\n", "1. **CCXT Library Setup**: How to install and configure CCXT for Binance\n", "2. **Secure API Integration**: Proper use of environment variables and dotenv for Binance API keys\n", "3. **Binance Connectivity**: Establishing authenticated connections with proper error handling\n", "4. **Data Fetching**: Retrieving ticker, OHLCV, and order book data\n", "5. **Rate Limiting**: Implementing rate limits to avoid API restrictions\n", "6. **Error Management**: Handling network errors, rate limits, and exchange downtime\n", "7. **Reconnection Logic**: Building robust connections that auto-recover\n", "8. **Market Analysis**: Advanced data processing and technical analysis\n", "\n", "### Next Steps:\n", "- Complete the data pipeline exercise\n", "- Move to Session 3.2 for technical indicators\n", "- Practice with different timeframes and symbols\n", "- Explore authenticated endpoints like trading history and account info\n", "- Learn about WebSocket connections for real-time data\n", "\n", "### Key Takeaways:\n", "- Always implement proper error handling for production systems\n", "- Rate limiting is crucial for maintaining API access\n", "- Connection monitoring prevents silent failures\n", "- Secure credential management is essential\n", "- Binance offers extensive market data for analysis\n", "- CCXT provides a standardized interface for cryptocurrency exchanges\n", "\n", "### Binance API Resources:\n", "- [Binance API Documentation](https://binance-docs.github.io/apidocs/spot/en/)\n", "- [CCXT Documentation](https://ccxt.readthedocs.io/en/latest/)\n", "- [Binance API Management](https://www.binance.com/en/my/settings/api-management)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 4 }