1212 lines
43 KiB
Text
1212 lines
43 KiB
Text
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Session 3.1: Binance API with CCXT\n",
|
||
"## Data Processing & Technical Analysis Foundation\n",
|
||
"\n",
|
||
"### Learning Objectives:\n",
|
||
"- Set up and configure CCXT library for Binance\n",
|
||
"- Connect to Binance with API authentication\n",
|
||
"- Handle API rate limits and errors\n",
|
||
"- Implement reconnection logic and fail-safes\n",
|
||
"- Fetch and process real-time market data\n",
|
||
"\n",
|
||
"### Prerequisites:\n",
|
||
"- Python 3.7+\n",
|
||
"- pandas, numpy\n",
|
||
"- Basic understanding of APIs\n",
|
||
"- Binance API keys (for authenticated endpoints)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 1. Setup and Environment Configuration"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 95,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Import necessary libraries\n",
|
||
"import ccxt\n",
|
||
"import pandas as pd\n",
|
||
"import numpy as np\n",
|
||
"import time\n",
|
||
"import logging\n",
|
||
"import os\n",
|
||
"from datetime import datetime, timedelta\n",
|
||
"from dotenv import load_dotenv\n",
|
||
"import warnings\n",
|
||
"warnings.filterwarnings('ignore')\n",
|
||
"\n",
|
||
"# Configure logging\n",
|
||
"logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 2. Secure API Key Management with dotenv\n",
|
||
"\n",
|
||
"### Why Use Environment Variables?\n",
|
||
"- **Security**: API keys should never be hardcoded in source code\n",
|
||
"- **Flexibility**: Easy to switch between different API keys\n",
|
||
"- **Version Control Safety**: .env files are excluded from Git repositories\n",
|
||
"- **Team Collaboration**: Each developer can have their own credentials\n",
|
||
"\n",
|
||
"### Setting Up dotenv\n",
|
||
"The `python-dotenv` library allows us to load environment variables from a `.env` file.\n",
|
||
"\n",
|
||
"**Installation**: `pip install python-dotenv`"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 96,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"🔐 Environment Variables Setup\n",
|
||
"==============================\n",
|
||
"📄 .env file exists: ❌ No\n",
|
||
"\n",
|
||
"📝 Create a .env file with the following structure:\n",
|
||
"\n",
|
||
"# .env file template\n",
|
||
"# Copy this to a new file named '.env' in your project directory\n",
|
||
"\n",
|
||
"# Binance API Keys (get from https://www.binance.com/en/my/settings/api-management)\n",
|
||
"BINANCE_API_KEY=your_binance_api_key_here\n",
|
||
"BINANCE_SECRET_KEY=your_binance_secret_key_here\n",
|
||
"\n",
|
||
"# General Settings\n",
|
||
"ENABLE_RATE_LIMIT=true\n",
|
||
"REQUEST_TIMEOUT=30000\n",
|
||
" \n",
|
||
"\n",
|
||
"⚠️ IMPORTANT SECURITY NOTES:\n",
|
||
"1. Add '.env' to your .gitignore file\n",
|
||
"2. Never commit API keys to version control\n",
|
||
"3. Use read-only permissions for learning/analysis\n",
|
||
"4. Regularly rotate your API keys\n",
|
||
"5. Limit API key permissions to minimum required\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Load environment variables from .env file\n",
|
||
"load_dotenv()\n",
|
||
"\n",
|
||
"# Verify dotenv is working\n",
|
||
"print(\"🔐 Environment Variables Setup\")\n",
|
||
"print(\"=\" * 30)\n",
|
||
"\n",
|
||
"# Check if .env file exists\n",
|
||
"env_file_exists = os.path.exists('.env')\n",
|
||
"print(f\"📄 .env file exists: {'✅ Yes' if env_file_exists else '❌ No'}\")\n",
|
||
"\n",
|
||
"if not env_file_exists:\n",
|
||
" print(\"\\n📝 Create a .env file with the following structure:\")\n",
|
||
" print(\"\"\"\n",
|
||
"# .env file template\n",
|
||
"# Copy this to a new file named '.env' in your project directory\n",
|
||
"\n",
|
||
"# Binance API Keys (get from https://www.binance.com/en/my/settings/api-management)\n",
|
||
"BINANCE_API_KEY=your_binance_api_key_here\n",
|
||
"BINANCE_SECRET_KEY=your_binance_secret_key_here\n",
|
||
"\n",
|
||
"# General Settings\n",
|
||
"ENABLE_RATE_LIMIT=true\n",
|
||
"REQUEST_TIMEOUT=30000\n",
|
||
" \"\"\")\n",
|
||
"else:\n",
|
||
" print(\"\\n🔍 Checking available environment variables:\")\n",
|
||
" # Check for Binance API key variables (without exposing values)\n",
|
||
" env_vars_to_check = [\n",
|
||
" 'BINANCE_API_KEY', 'BINANCE_SECRET_KEY', 'ENABLE_RATE_LIMIT'\n",
|
||
" ]\n",
|
||
" \n",
|
||
" for var in env_vars_to_check:\n",
|
||
" value = os.getenv(var)\n",
|
||
" if value:\n",
|
||
" # Show first 4 and last 4 characters for API keys, or full value for settings\n",
|
||
" if 'API_KEY' in var or 'SECRET' in var:\n",
|
||
" display_value = f\"{value[:4]}...{value[-4:]}\" if len(value) > 8 else \"***\"\n",
|
||
" else:\n",
|
||
" display_value = value\n",
|
||
" print(f\" ✅ {var}: {display_value}\")\n",
|
||
" else:\n",
|
||
" print(f\" ❌ {var}: Not set\")\n",
|
||
"\n",
|
||
"print(\"\\n⚠️ IMPORTANT SECURITY NOTES:\")\n",
|
||
"print(\"1. Add '.env' to your .gitignore file\")\n",
|
||
"print(\"2. Never commit API keys to version control\")\n",
|
||
"print(\"3. Use read-only permissions for learning/analysis\")\n",
|
||
"print(\"4. Regularly rotate your API keys\")\n",
|
||
"print(\"5. Limit API key permissions to minimum required\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Creating Your .env File\n",
|
||
"\n",
|
||
"**Step 1**: Create a file named `.env` in your project directory\n",
|
||
"\n",
|
||
"**Step 2**: Add your Binance API credentials:\n",
|
||
"\n",
|
||
"```bash\n",
|
||
"# Binance Configuration\n",
|
||
"BINANCE_API_KEY=your_actual_binance_api_key_here\n",
|
||
"BINANCE_SECRET_KEY=your_actual_binance_secret_key_here\n",
|
||
"\n",
|
||
"# Settings\n",
|
||
"ENABLE_RATE_LIMIT=true\n",
|
||
"REQUEST_TIMEOUT=30000\n",
|
||
"```\n",
|
||
"\n",
|
||
"### Getting Binance API Keys\n",
|
||
"\n",
|
||
"1. **Login** to your Binance account\n",
|
||
"2. **Navigate** to API Management: Account → API Management\n",
|
||
"3. **Create** a new API key with a descriptive label\n",
|
||
"4. **Set Permissions**: For learning, enable only:\n",
|
||
" - ✅ Read Info (spot & margin)\n",
|
||
" - ❌ Spot & Margin Trading (disable for safety)\n",
|
||
" - ❌ Futures (disable for safety)\n",
|
||
" - ❌ Withdrawals (never enable for learning)\n",
|
||
"5. **IP Restrictions**: Add your IP address for extra security\n",
|
||
"6. **Copy** both the API Key and Secret Key to your .env file\n",
|
||
"\n",
|
||
"### API Key Security Best Practices\n",
|
||
"\n",
|
||
"1. **Minimal Permissions**: Only grant read permissions for learning\n",
|
||
"2. **IP Restrictions**: Restrict API access to your specific IP address\n",
|
||
"3. **Key Rotation**: Regularly rotate your API keys (monthly recommended)\n",
|
||
"4. **Monitor Usage**: Check API usage logs regularly for unusual activity\n",
|
||
"5. **Never Share**: Never share API keys in chat, email, or code repositories"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 3. CCXT Library Overview"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 97,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"CCXT Version: 4.4.85\n",
|
||
"Total Supported Exchanges: 105\n",
|
||
"\n",
|
||
"Binance Support: ✅ Yes\n",
|
||
"\n",
|
||
"Other Popular Exchanges Supported by CCXT:\n",
|
||
" ✓ coinbase\n",
|
||
" ✓ kraken\n",
|
||
" ✓ bitfinex\n",
|
||
" ✓ huobi\n",
|
||
" ✓ okx\n",
|
||
" ✓ kucoin\n",
|
||
" ✓ bybit\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Check CCXT version and Binance support\n",
|
||
"print(f\"CCXT Version: {ccxt.__version__}\")\n",
|
||
"print(f\"Total Supported Exchanges: {len(ccxt.exchanges)}\")\n",
|
||
"print(f\"\\nBinance Support: {'✅ Yes' if 'binance' in ccxt.exchanges else '❌ No'}\")\n",
|
||
"\n",
|
||
"# Show some popular exchanges that CCXT supports\n",
|
||
"print(\"\\nOther Popular Exchanges Supported by CCXT:\")\n",
|
||
"popular_exchanges = ['coinbase', 'kraken', 'bitfinex', 'huobi', 'okx', 'kucoin', 'bybit']\n",
|
||
"for exchange in popular_exchanges:\n",
|
||
" if exchange in ccxt.exchanges:\n",
|
||
" print(f\" ✓ {exchange}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 4. Connecting to Binance with API Authentication"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 98,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"2025-06-18 20:09:06,681 - WARNING - No Binance credentials found, using public API only\n",
|
||
"2025-06-18 20:09:09,288 - INFO - Successfully connected to Binance\n",
|
||
"2025-06-18 20:09:09,288 - INFO - Available markets: 3730\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"def get_binance_credentials():\n",
|
||
" \"\"\"\n",
|
||
" Retrieve Binance API credentials from environment variables\n",
|
||
" \"\"\"\n",
|
||
" api_key = os.getenv('BINANCE_API_KEY')\n",
|
||
" secret = os.getenv('BINANCE_SECRET_KEY')\n",
|
||
" \n",
|
||
" if api_key and secret:\n",
|
||
" return {\n",
|
||
" 'apiKey': api_key,\n",
|
||
" 'secret': secret,\n",
|
||
" }\n",
|
||
" return {}\n",
|
||
"\n",
|
||
"def initialize_binance(use_credentials=True):\n",
|
||
" \"\"\"\n",
|
||
" Initialize Binance connection with optional API authentication\n",
|
||
" \"\"\"\n",
|
||
" try:\n",
|
||
" # Base configuration\n",
|
||
" config = {\n",
|
||
" 'enableRateLimit': True, # Enable built-in rate limiting\n",
|
||
" 'timeout': int(os.getenv('REQUEST_TIMEOUT', 30000)), # 30 seconds timeout\n",
|
||
" }\n",
|
||
" \n",
|
||
" # Add API credentials if available and requested\n",
|
||
" if use_credentials:\n",
|
||
" credentials = get_binance_credentials()\n",
|
||
" if credentials:\n",
|
||
" config.update(credentials)\n",
|
||
" logging.info(\"Using authenticated connection for Binance\")\n",
|
||
" else:\n",
|
||
" logging.warning(\"No Binance credentials found, using public API only\")\n",
|
||
" \n",
|
||
" # Initialize Binance exchange\n",
|
||
" binance = ccxt.binance(config)\n",
|
||
" \n",
|
||
" # Test connection\n",
|
||
" markets = binance.load_markets()\n",
|
||
" logging.info(f\"Successfully connected to Binance\")\n",
|
||
" logging.info(f\"Available markets: {len(markets)}\")\n",
|
||
" \n",
|
||
" # Test authentication if credentials were provided\n",
|
||
" if use_credentials and config.get('apiKey'):\n",
|
||
" try:\n",
|
||
" # Try to fetch account info (requires authentication)\n",
|
||
" account = binance.fetch_balance()\n",
|
||
" logging.info(\"✅ Authentication successful for Binance\")\n",
|
||
" except ccxt.AuthenticationError:\n",
|
||
" logging.error(\"❌ Authentication failed for Binance - check your API keys\")\n",
|
||
" except ccxt.PermissionDenied:\n",
|
||
" logging.warning(\"⚠️ Limited permissions for Binance - some features may not work\")\n",
|
||
" except Exception as e:\n",
|
||
" logging.warning(f\"Could not test authentication for Binance: {str(e)}\")\n",
|
||
" \n",
|
||
" return binance\n",
|
||
" \n",
|
||
" except Exception as e:\n",
|
||
" logging.error(f\"Failed to connect to Binance: {str(e)}\")\n",
|
||
" return None\n",
|
||
"\n",
|
||
"# Initialize Binance connection\n",
|
||
"binance = initialize_binance(use_credentials=True)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 5. Market Data Exploration"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 99,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"BTC Trading pairs (first 10): ['BTC/USDT', 'BTC/USDT:USDT', 'BTC/USDT:USDT-250627', 'BTC/USDT:USDT-250926', 'BTCDOM/USDT:USDT', 'BTCDOWN/USDT', 'BTCST/USDT', 'BTCST/USDT:USDT', 'BTCUP/USDT', 'PUMPBTC/USDT:USDT']\n",
|
||
"\n",
|
||
"Available popular pairs: ['BTC/USDT', 'ETH/USDT', 'BNB/USDT', 'ADA/USDT', 'SOL/USDT', 'DOT/USDT']\n",
|
||
"\n",
|
||
"BTC/USDT Market Info:\n",
|
||
" Base: BTC\n",
|
||
" Quote: USDT\n",
|
||
" Active: True\n",
|
||
" Spot: True\n",
|
||
" Maker Fee: 0.001\n",
|
||
" Taker Fee: 0.001\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Explore available trading pairs on Binance\n",
|
||
"if binance:\n",
|
||
" # Get BTC trading pairs\n",
|
||
" btc_pairs = [symbol for symbol in binance.symbols if 'BTC' in symbol and '/USDT' in symbol]\n",
|
||
" print(f\"BTC Trading pairs (first 10): {btc_pairs[:10]}\")\n",
|
||
" \n",
|
||
" # Get popular trading pairs\n",
|
||
" popular_pairs = ['BTC/USDT', 'ETH/USDT', 'BNB/USDT', 'ADA/USDT', 'SOL/USDT', 'DOT/USDT']\n",
|
||
" available_pairs = [pair for pair in popular_pairs if pair in binance.symbols]\n",
|
||
" print(f\"\\nAvailable popular pairs: {available_pairs}\")\n",
|
||
" \n",
|
||
" # Show market info for BTC/USDT\n",
|
||
" btc_market = binance.market('BTC/USDT')\n",
|
||
" print(f\"\\nBTC/USDT Market Info:\")\n",
|
||
" print(f\" Base: {btc_market['base']}\")\n",
|
||
" print(f\" Quote: {btc_market['quote']}\")\n",
|
||
" print(f\" Active: {btc_market['active']}\")\n",
|
||
" print(f\" Spot: {btc_market['spot']}\")\n",
|
||
" print(f\" Maker Fee: {btc_market['maker']}\")\n",
|
||
" print(f\" Taker Fee: {btc_market['taker']}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 6. Basic Data Fetching"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 100,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Current Market Prices:\n",
|
||
"========================================\n",
|
||
"BTC/USDT: $104,165.23 (+0.28%)\n",
|
||
" Volume: 13,936.07 BTC\n",
|
||
" Bid/Ask: $104,165.22 / $104,165.23\n",
|
||
"\n",
|
||
"ETH/USDT: $2,488.32 (+0.69%)\n",
|
||
" Volume: 523,024.45 ETH\n",
|
||
" Bid/Ask: $2,488.31 / $2,488.32\n",
|
||
"\n",
|
||
"BNB/USDT: $640.48 (-0.36%)\n",
|
||
" Volume: 141,740.24 BNB\n",
|
||
" Bid/Ask: $640.48 / $640.49\n",
|
||
"\n",
|
||
"ADA/USDT: $0.59 (-2.16%)\n",
|
||
" Volume: 101,214,201.90 ADA\n",
|
||
" Bid/Ask: $0.59 / $0.59\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"def fetch_ticker(symbol):\n",
|
||
" \"\"\"\n",
|
||
" Fetch current ticker data with error handling\n",
|
||
" \"\"\"\n",
|
||
" try:\n",
|
||
" ticker = binance.fetch_ticker(symbol)\n",
|
||
" return {\n",
|
||
" 'symbol': symbol,\n",
|
||
" 'last': ticker['last'],\n",
|
||
" 'bid': ticker['bid'],\n",
|
||
" 'ask': ticker['ask'],\n",
|
||
" 'volume': ticker['baseVolume'],\n",
|
||
" 'change': ticker['change'],\n",
|
||
" 'percentage': ticker['percentage'],\n",
|
||
" 'timestamp': datetime.fromtimestamp(ticker['timestamp'] / 1000)\n",
|
||
" }\n",
|
||
" except Exception as e:\n",
|
||
" logging.error(f\"Error fetching ticker for {symbol}: {str(e)}\")\n",
|
||
" return None\n",
|
||
"\n",
|
||
"# Fetch ticker data for popular pairs\n",
|
||
"if binance:\n",
|
||
" print(\"Current Market Prices:\")\n",
|
||
" print(\"=\" * 40)\n",
|
||
" \n",
|
||
" for symbol in ['BTC/USDT', 'ETH/USDT', 'BNB/USDT', 'ADA/USDT']:\n",
|
||
" ticker_data = fetch_ticker(symbol)\n",
|
||
" if ticker_data:\n",
|
||
" print(f\"{symbol}: ${ticker_data['last']:,.2f} ({ticker_data['percentage']:+.2f}%)\")\n",
|
||
" print(f\" Volume: {ticker_data['volume']:,.2f} {symbol.split('/')[0]}\")\n",
|
||
" print(f\" Bid/Ask: ${ticker_data['bid']:,.2f} / ${ticker_data['ask']:,.2f}\")\n",
|
||
" print()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 7. OHLCV Data Fetching"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 101,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"2025-06-18 20:09:10,790 - INFO - Fetched 100 candles for BTC/USDT (1h)\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"BTC/USDT OHLCV Data (last 5 candles):\n",
|
||
" open high low close volume\n",
|
||
"datetime \n",
|
||
"2025-06-18 14:00:00 104439.82 105325.83 104228.30 104619.88 1418.42072\n",
|
||
"2025-06-18 15:00:00 104619.87 104973.79 104150.13 104801.01 869.24120\n",
|
||
"2025-06-18 16:00:00 104801.01 104836.64 104334.78 104426.93 509.29545\n",
|
||
"2025-06-18 17:00:00 104426.93 104508.00 103615.33 104331.83 918.22252\n",
|
||
"2025-06-18 18:00:00 104332.00 104706.08 104011.42 104165.23 265.75381\n",
|
||
"\n",
|
||
"Data shape: (100, 5)\n",
|
||
"Date range: 2025-06-14 15:00:00 to 2025-06-18 18:00:00\n",
|
||
"\n",
|
||
"Price Statistics:\n",
|
||
" Highest: $108,952.38\n",
|
||
" Lowest: $103,371.02\n",
|
||
" Average Volume: 539.80 BTC\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"def fetch_ohlcv_data(symbol, timeframe='1h', limit=100):\n",
|
||
" \"\"\"\n",
|
||
" Fetch OHLCV data with error handling and data processing\n",
|
||
" \"\"\"\n",
|
||
" try:\n",
|
||
" # Fetch raw OHLCV data\n",
|
||
" ohlcv = binance.fetch_ohlcv(symbol, timeframe, limit=limit)\n",
|
||
" \n",
|
||
" # Convert to DataFrame\n",
|
||
" df = pd.DataFrame(ohlcv, columns=['timestamp', 'open', 'high', 'low', 'close', 'volume'])\n",
|
||
" df['datetime'] = pd.to_datetime(df['timestamp'], unit='ms')\n",
|
||
" df.set_index('datetime', inplace=True)\n",
|
||
" df.drop('timestamp', axis=1, inplace=True)\n",
|
||
" \n",
|
||
" # Ensure numeric types\n",
|
||
" for col in ['open', 'high', 'low', 'close', 'volume']:\n",
|
||
" df[col] = pd.to_numeric(df[col], errors='coerce')\n",
|
||
" \n",
|
||
" logging.info(f\"Fetched {len(df)} candles for {symbol} ({timeframe})\")\n",
|
||
" return df\n",
|
||
" \n",
|
||
" except Exception as e:\n",
|
||
" logging.error(f\"Error fetching OHLCV data for {symbol}: {str(e)}\")\n",
|
||
" return None\n",
|
||
"\n",
|
||
"# Fetch OHLCV data example\n",
|
||
"if binance:\n",
|
||
" btc_data = fetch_ohlcv_data('BTC/USDT', '1h', 100)\n",
|
||
" if btc_data is not None:\n",
|
||
" print(\"BTC/USDT OHLCV Data (last 5 candles):\")\n",
|
||
" print(btc_data.tail())\n",
|
||
" print(f\"\\nData shape: {btc_data.shape}\")\n",
|
||
" print(f\"Date range: {btc_data.index.min()} to {btc_data.index.max()}\")\n",
|
||
" \n",
|
||
" # Show basic statistics\n",
|
||
" print(f\"\\nPrice Statistics:\")\n",
|
||
" print(f\" Highest: ${btc_data['high'].max():,.2f}\")\n",
|
||
" print(f\" Lowest: ${btc_data['low'].min():,.2f}\")\n",
|
||
" print(f\" Average Volume: {btc_data['volume'].mean():,.2f} BTC\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 8. Rate Limiting and Error Management"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 102,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Testing robust data fetching...\n",
|
||
"Fetching BTC/USDT...\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"2025-06-18 20:09:11,174 - INFO - Fetched 50 candles for BTC/USDT (1h)\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"✓ BTC/USDT: 50 candles\n",
|
||
"Fetching ETH/USDT...\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"2025-06-18 20:09:12,484 - INFO - Fetched 50 candles for ETH/USDT (1h)\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"✓ ETH/USDT: 50 candles\n",
|
||
"Fetching BNB/USDT...\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"2025-06-18 20:09:13,869 - INFO - Fetched 50 candles for BNB/USDT (1h)\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"✓ BNB/USDT: 50 candles\n",
|
||
"Fetching ADA/USDT...\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"2025-06-18 20:09:15,178 - INFO - Fetched 50 candles for ADA/USDT (1h)\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"✓ ADA/USDT: 50 candles\n",
|
||
"\n",
|
||
"Total time: 4.38 seconds\n",
|
||
"Successfully fetched data for 4 symbols\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"import time\n",
|
||
"from functools import wraps\n",
|
||
"\n",
|
||
"def rate_limit_decorator(calls_per_second=1):\n",
|
||
" \"\"\"\n",
|
||
" Decorator to implement rate limiting\n",
|
||
" \"\"\"\n",
|
||
" min_interval = 1.0 / calls_per_second\n",
|
||
" last_called = [0.0]\n",
|
||
" \n",
|
||
" def decorator(func):\n",
|
||
" @wraps(func)\n",
|
||
" def wrapper(*args, **kwargs):\n",
|
||
" elapsed = time.time() - last_called[0]\n",
|
||
" left_to_wait = min_interval - elapsed\n",
|
||
" if left_to_wait > 0:\n",
|
||
" time.sleep(left_to_wait)\n",
|
||
" ret = func(*args, **kwargs)\n",
|
||
" last_called[0] = time.time()\n",
|
||
" return ret\n",
|
||
" return wrapper\n",
|
||
" return decorator\n",
|
||
"\n",
|
||
"def exponential_backoff(max_retries=3, base_delay=1):\n",
|
||
" \"\"\"\n",
|
||
" Decorator for exponential backoff retry logic\n",
|
||
" \"\"\"\n",
|
||
" def decorator(func):\n",
|
||
" @wraps(func)\n",
|
||
" def wrapper(*args, **kwargs):\n",
|
||
" for attempt in range(max_retries + 1):\n",
|
||
" try:\n",
|
||
" return func(*args, **kwargs)\n",
|
||
" except ccxt.NetworkError as e:\n",
|
||
" if attempt == max_retries:\n",
|
||
" logging.error(f\"Max retries reached for {func.__name__}: {str(e)}\")\n",
|
||
" raise\n",
|
||
" delay = base_delay * (2 ** attempt)\n",
|
||
" logging.warning(f\"Network error in {func.__name__}, retrying in {delay}s... (attempt {attempt + 1}/{max_retries + 1})\")\n",
|
||
" time.sleep(delay)\n",
|
||
" except ccxt.RateLimitExceeded as e:\n",
|
||
" if attempt == max_retries:\n",
|
||
" logging.error(f\"Rate limit exceeded for {func.__name__}: {str(e)}\")\n",
|
||
" raise\n",
|
||
" delay = base_delay * (4 ** attempt) # Longer delay for rate limits\n",
|
||
" logging.warning(f\"Rate limit exceeded in {func.__name__}, waiting {delay}s... (attempt {attempt + 1}/{max_retries + 1})\")\n",
|
||
" time.sleep(delay)\n",
|
||
" except Exception as e:\n",
|
||
" logging.error(f\"Unexpected error in {func.__name__}: {str(e)}\")\n",
|
||
" raise\n",
|
||
" return None\n",
|
||
" return wrapper\n",
|
||
" return decorator\n",
|
||
"\n",
|
||
"# Enhanced data fetching with error handling\n",
|
||
"@exponential_backoff(max_retries=3, base_delay=1)\n",
|
||
"@rate_limit_decorator(calls_per_second=1)\n",
|
||
"def robust_fetch_ohlcv(symbol, timeframe='1h', limit=100):\n",
|
||
" \"\"\"\n",
|
||
" Robust OHLCV fetching with rate limiting and error handling\n",
|
||
" \"\"\"\n",
|
||
" return fetch_ohlcv_data(symbol, timeframe, limit)\n",
|
||
"\n",
|
||
"# Test robust fetching\n",
|
||
"if binance:\n",
|
||
" print(\"Testing robust data fetching...\")\n",
|
||
" start_time = time.time()\n",
|
||
" \n",
|
||
" # Fetch data for multiple symbols\n",
|
||
" symbols = ['BTC/USDT', 'ETH/USDT', 'BNB/USDT', 'ADA/USDT']\n",
|
||
" data = {}\n",
|
||
" \n",
|
||
" for symbol in symbols:\n",
|
||
" print(f\"Fetching {symbol}...\")\n",
|
||
" df = robust_fetch_ohlcv(symbol, '1h', 50)\n",
|
||
" if df is not None:\n",
|
||
" data[symbol] = df\n",
|
||
" print(f\"✓ {symbol}: {len(df)} candles\")\n",
|
||
" else:\n",
|
||
" print(f\"✗ Failed to fetch {symbol}\")\n",
|
||
" \n",
|
||
" end_time = time.time()\n",
|
||
" print(f\"\\nTotal time: {end_time - start_time:.2f} seconds\")\n",
|
||
" print(f\"Successfully fetched data for {len(data)} symbols\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 9. Reconnection Logic and Connection Monitoring"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 103,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Testing BinanceManager...\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"2025-06-18 20:09:17,482 - INFO - Connected to Binance\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"✓ Binance manager is healthy\n",
|
||
"✓ BTC/USDT: $104,164.51\n",
|
||
"✓ ETH/USDT OHLCV: 10 candles\n",
|
||
"✓ BTC/USDT Order Book: 5 bids, 5 asks\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"class BinanceManager:\n",
|
||
" \"\"\"\n",
|
||
" Binance manager with connection monitoring and auto-reconnection\n",
|
||
" \"\"\"\n",
|
||
" \n",
|
||
" def __init__(self, use_credentials=True):\n",
|
||
" self.use_credentials = use_credentials\n",
|
||
" self.exchange = None\n",
|
||
" self.last_successful_call = None\n",
|
||
" self.connection_failures = 0\n",
|
||
" self.max_failures = 5\n",
|
||
" \n",
|
||
" self._connect()\n",
|
||
" \n",
|
||
" def _connect(self):\n",
|
||
" \"\"\"\n",
|
||
" Establish connection to Binance\n",
|
||
" \"\"\"\n",
|
||
" try:\n",
|
||
" # Base configuration\n",
|
||
" config = {\n",
|
||
" 'enableRateLimit': True,\n",
|
||
" 'timeout': int(os.getenv('REQUEST_TIMEOUT', 30000)),\n",
|
||
" }\n",
|
||
" \n",
|
||
" # Add credentials if requested\n",
|
||
" if self.use_credentials:\n",
|
||
" credentials = get_binance_credentials()\n",
|
||
" if credentials:\n",
|
||
" config.update(credentials)\n",
|
||
" \n",
|
||
" self.exchange = ccxt.binance(config)\n",
|
||
" \n",
|
||
" # Test connection\n",
|
||
" self.exchange.load_markets()\n",
|
||
" self.connection_failures = 0\n",
|
||
" self.last_successful_call = datetime.now()\n",
|
||
" \n",
|
||
" logging.info(f\"Connected to Binance\")\n",
|
||
" return True\n",
|
||
" \n",
|
||
" except Exception as e:\n",
|
||
" logging.error(f\"Failed to connect to Binance: {str(e)}\")\n",
|
||
" self.connection_failures += 1\n",
|
||
" return False\n",
|
||
" \n",
|
||
" def _reconnect_if_needed(self):\n",
|
||
" \"\"\"\n",
|
||
" Reconnect if connection seems stale or failed\n",
|
||
" \"\"\"\n",
|
||
" if (self.connection_failures >= self.max_failures or \n",
|
||
" self.last_successful_call is None or \n",
|
||
" (datetime.now() - self.last_successful_call).seconds > 300): # 5 minutes\n",
|
||
" \n",
|
||
" logging.warning(f\"Reconnecting to Binance...\")\n",
|
||
" return self._connect()\n",
|
||
" return True\n",
|
||
" \n",
|
||
" def execute_with_retry(self, func, *args, **kwargs):\n",
|
||
" \"\"\"\n",
|
||
" Execute exchange function with automatic retry and reconnection\n",
|
||
" \"\"\"\n",
|
||
" max_retries = 3\n",
|
||
" \n",
|
||
" for attempt in range(max_retries):\n",
|
||
" try:\n",
|
||
" # Check if reconnection is needed\n",
|
||
" if not self._reconnect_if_needed():\n",
|
||
" continue\n",
|
||
" \n",
|
||
" # Execute the function\n",
|
||
" result = func(*args, **kwargs)\n",
|
||
" \n",
|
||
" # Update success timestamp\n",
|
||
" self.last_successful_call = datetime.now()\n",
|
||
" self.connection_failures = 0\n",
|
||
" \n",
|
||
" return result\n",
|
||
" \n",
|
||
" except (ccxt.NetworkError, ccxt.ExchangeNotAvailable) as e:\n",
|
||
" self.connection_failures += 1\n",
|
||
" logging.warning(f\"Network/Exchange error (attempt {attempt + 1}): {str(e)}\")\n",
|
||
" \n",
|
||
" if attempt < max_retries - 1:\n",
|
||
" time.sleep(2 ** attempt) # Exponential backoff\n",
|
||
" \n",
|
||
" except ccxt.RateLimitExceeded as e:\n",
|
||
" logging.warning(f\"Rate limit exceeded (attempt {attempt + 1}): {str(e)}\")\n",
|
||
" \n",
|
||
" if attempt < max_retries - 1:\n",
|
||
" time.sleep(5 * (attempt + 1)) # Longer wait for rate limits\n",
|
||
" \n",
|
||
" except Exception as e:\n",
|
||
" logging.error(f\"Unexpected error: {str(e)}\")\n",
|
||
" break\n",
|
||
" \n",
|
||
" logging.error(f\"Failed to execute function after {max_retries} attempts\")\n",
|
||
" return None\n",
|
||
" \n",
|
||
" def fetch_ticker(self, symbol):\n",
|
||
" return self.execute_with_retry(self.exchange.fetch_ticker, symbol)\n",
|
||
" \n",
|
||
" def fetch_ohlcv(self, symbol, timeframe='1h', limit=100):\n",
|
||
" return self.execute_with_retry(self.exchange.fetch_ohlcv, symbol, timeframe, limit=limit)\n",
|
||
" \n",
|
||
" def fetch_balance(self):\n",
|
||
" \"\"\"Fetch account balance (requires authentication)\"\"\"\n",
|
||
" return self.execute_with_retry(self.exchange.fetch_balance)\n",
|
||
" \n",
|
||
" def fetch_order_book(self, symbol, limit=100):\n",
|
||
" \"\"\"Fetch order book for a symbol\"\"\"\n",
|
||
" return self.execute_with_retry(self.exchange.fetch_order_book, symbol, limit)\n",
|
||
" \n",
|
||
" def is_healthy(self):\n",
|
||
" \"\"\"\n",
|
||
" Check if connection is healthy\n",
|
||
" \"\"\"\n",
|
||
" return (self.connection_failures < self.max_failures and \n",
|
||
" self.last_successful_call is not None and \n",
|
||
" (datetime.now() - self.last_successful_call).seconds < 300)\n",
|
||
"\n",
|
||
"# Test the BinanceManager\n",
|
||
"print(\"Testing BinanceManager...\")\n",
|
||
"manager = BinanceManager(use_credentials=True)\n",
|
||
"\n",
|
||
"if manager.is_healthy():\n",
|
||
" print(\"✓ Binance manager is healthy\")\n",
|
||
" \n",
|
||
" # Test ticker fetching\n",
|
||
" ticker = manager.fetch_ticker('BTC/USDT')\n",
|
||
" if ticker:\n",
|
||
" print(f\"✓ BTC/USDT: ${ticker['last']:,.2f}\")\n",
|
||
" \n",
|
||
" # Test OHLCV fetching\n",
|
||
" ohlcv = manager.fetch_ohlcv('ETH/USDT', '1h', 10)\n",
|
||
" if ohlcv:\n",
|
||
" print(f\"✓ ETH/USDT OHLCV: {len(ohlcv)} candles\")\n",
|
||
" \n",
|
||
" # Test order book fetching\n",
|
||
" order_book = manager.fetch_order_book('BTC/USDT', 5)\n",
|
||
" if order_book:\n",
|
||
" print(f\"✓ BTC/USDT Order Book: {len(order_book['bids'])} bids, {len(order_book['asks'])} asks\")\n",
|
||
" \n",
|
||
" # Test authenticated endpoint (if credentials are available)\n",
|
||
" if manager.use_credentials and manager.exchange.apiKey:\n",
|
||
" try:\n",
|
||
" balance = manager.fetch_balance()\n",
|
||
" if balance:\n",
|
||
" print(\"✓ Balance fetched successfully (authenticated)\")\n",
|
||
" except Exception as e:\n",
|
||
" print(f\"⚠️ Could not fetch balance: {str(e)}\")\n",
|
||
"else:\n",
|
||
" print(\"✗ Binance manager is not healthy\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 10. Advanced Data Analysis"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 104,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Market Analysis Report\n",
|
||
"==================================================\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"2025-06-18 20:09:19,264 - INFO - Fetched 100 candles for BTC/USDT (1h)\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"BTC/USDT Analysis:\n",
|
||
" Current Price: $104,149.50\n",
|
||
" 24h Change: -0.25%\n",
|
||
" Price Range: $103,371.02 - $108,952.38\n",
|
||
" Average Volume: 539.81\n",
|
||
" Volatility: 1.33%\n",
|
||
" SMA 20: $104,752.94\n",
|
||
" SMA 50: $105,660.09\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"2025-06-18 20:09:19,585 - INFO - Fetched 100 candles for ETH/USDT (1h)\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"ETH/USDT Analysis:\n",
|
||
" Current Price: $2,487.00\n",
|
||
" 24h Change: -0.37%\n",
|
||
" Price Range: $2,453.80 - $2,680.34\n",
|
||
" Average Volume: 22,377.63\n",
|
||
" Volatility: 2.53%\n",
|
||
" SMA 20: $2,514.66\n",
|
||
" SMA 50: $2,545.11\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"2025-06-18 20:09:19,872 - INFO - Fetched 100 candles for BNB/USDT (1h)\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"BNB/USDT Analysis:\n",
|
||
" Current Price: $640.38\n",
|
||
" 24h Change: -0.88%\n",
|
||
" Price Range: $637.35 - $659.46\n",
|
||
" Average Volume: 5,546.83\n",
|
||
" Volatility: 1.53%\n",
|
||
" SMA 20: $646.92\n",
|
||
" SMA 50: $650.47\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"def analyze_market_data(symbol, timeframe='1h', limit=100):\n",
|
||
" \"\"\"\n",
|
||
" Comprehensive market data analysis\n",
|
||
" \"\"\"\n",
|
||
" if not binance:\n",
|
||
" print(\"Binance connection not available\")\n",
|
||
" return None\n",
|
||
" \n",
|
||
" try:\n",
|
||
" # Fetch OHLCV data\n",
|
||
" df = fetch_ohlcv_data(symbol, timeframe, limit)\n",
|
||
" if df is None:\n",
|
||
" return None\n",
|
||
" \n",
|
||
" # Calculate additional metrics\n",
|
||
" df['price_change'] = df['close'].pct_change()\n",
|
||
" df['high_low_range'] = df['high'] - df['low']\n",
|
||
" df['body_size'] = abs(df['close'] - df['open'])\n",
|
||
" df['upper_wick'] = df['high'] - df[['open', 'close']].max(axis=1)\n",
|
||
" df['lower_wick'] = df[['open', 'close']].min(axis=1) - df['low']\n",
|
||
" \n",
|
||
" # Calculate moving averages\n",
|
||
" df['sma_20'] = df['close'].rolling(window=20).mean()\n",
|
||
" df['sma_50'] = df['close'].rolling(window=50).mean()\n",
|
||
" \n",
|
||
" # Calculate volatility\n",
|
||
" df['volatility'] = df['price_change'].rolling(window=20).std() * np.sqrt(24) # 24-hour volatility\n",
|
||
" \n",
|
||
" # Analysis results\n",
|
||
" analysis = {\n",
|
||
" 'symbol': symbol,\n",
|
||
" 'timeframe': timeframe,\n",
|
||
" 'period': f\"{df.index.min()} to {df.index.max()}\",\n",
|
||
" 'current_price': df['close'].iloc[-1],\n",
|
||
" 'price_change_24h': df['price_change'].iloc[-24:].sum() if len(df) >= 24 else None,\n",
|
||
" 'highest_price': df['high'].max(),\n",
|
||
" 'lowest_price': df['low'].min(),\n",
|
||
" 'average_volume': df['volume'].mean(),\n",
|
||
" 'current_volatility': df['volatility'].iloc[-1] if not pd.isna(df['volatility'].iloc[-1]) else None,\n",
|
||
" 'sma_20': df['sma_20'].iloc[-1] if not pd.isna(df['sma_20'].iloc[-1]) else None,\n",
|
||
" 'sma_50': df['sma_50'].iloc[-1] if not pd.isna(df['sma_50'].iloc[-1]) else None,\n",
|
||
" }\n",
|
||
" \n",
|
||
" return analysis, df\n",
|
||
" \n",
|
||
" except Exception as e:\n",
|
||
" logging.error(f\"Error analyzing market data for {symbol}: {str(e)}\")\n",
|
||
" return None\n",
|
||
"\n",
|
||
"# Analyze multiple symbols\n",
|
||
"if binance:\n",
|
||
" print(\"Market Analysis Report\")\n",
|
||
" print(\"=\" * 50)\n",
|
||
" \n",
|
||
" symbols_to_analyze = ['BTC/USDT', 'ETH/USDT', 'BNB/USDT']\n",
|
||
" \n",
|
||
" for symbol in symbols_to_analyze:\n",
|
||
" result = analyze_market_data(symbol, '1h', 100)\n",
|
||
" if result:\n",
|
||
" analysis, df = result\n",
|
||
" \n",
|
||
" print(f\"\\n{symbol} Analysis:\")\n",
|
||
" print(f\" Current Price: ${analysis['current_price']:,.2f}\")\n",
|
||
" if analysis['price_change_24h']:\n",
|
||
" print(f\" 24h Change: {analysis['price_change_24h']:.2%}\")\n",
|
||
" print(f\" Price Range: ${analysis['lowest_price']:,.2f} - ${analysis['highest_price']:,.2f}\")\n",
|
||
" print(f\" Average Volume: {analysis['average_volume']:,.2f}\")\n",
|
||
" if analysis['current_volatility']:\n",
|
||
" print(f\" Volatility: {analysis['current_volatility']:.2%}\")\n",
|
||
" if analysis['sma_20']:\n",
|
||
" print(f\" SMA 20: ${analysis['sma_20']:,.2f}\")\n",
|
||
" if analysis['sma_50']:\n",
|
||
" print(f\" SMA 50: ${analysis['sma_50']:,.2f}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 11. Exercise: Build Your Data Pipeline"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 105,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Exercise: Create a data pipeline that:\n",
|
||
"# 1. Fetches data from Binance for multiple symbols\n",
|
||
"# 2. Handles errors gracefully\n",
|
||
"# 3. Stores data in a structured format\n",
|
||
"# 4. Monitors data quality\n",
|
||
"# 5. Provides summary statistics\n",
|
||
"\n",
|
||
"class BinanceDataPipeline:\n",
|
||
" def __init__(self, symbols, timeframe='1h', use_credentials=True):\n",
|
||
" self.symbols = symbols\n",
|
||
" self.timeframe = timeframe\n",
|
||
" self.manager = BinanceManager(use_credentials=use_credentials)\n",
|
||
" self.data_store = {}\n",
|
||
" self.quality_report = {}\n",
|
||
" \n",
|
||
" def fetch_all_data(self, limit=100):\n",
|
||
" \"\"\"\n",
|
||
" TODO: Implement data fetching for all symbols\n",
|
||
" - Fetch OHLCV data for each symbol\n",
|
||
" - Store results in self.data_store\n",
|
||
" - Handle errors for individual symbols\n",
|
||
" \"\"\"\n",
|
||
" pass\n",
|
||
" \n",
|
||
" def validate_data(self, symbol, df):\n",
|
||
" \"\"\"\n",
|
||
" TODO: Implement data quality checks\n",
|
||
" - Check for missing values\n",
|
||
" - Validate price ranges (no negative prices)\n",
|
||
" - Check for duplicate timestamps\n",
|
||
" - Validate volume data\n",
|
||
" - Return quality score and issues found\n",
|
||
" \"\"\"\n",
|
||
" pass\n",
|
||
" \n",
|
||
" def generate_summary(self):\n",
|
||
" \"\"\"\n",
|
||
" TODO: Generate summary statistics\n",
|
||
" - Calculate correlation between symbols\n",
|
||
" - Show price changes\n",
|
||
" - Volume analysis\n",
|
||
" - Data quality summary\n",
|
||
" \"\"\"\n",
|
||
" pass\n",
|
||
" \n",
|
||
" def run_pipeline(self, limit=100):\n",
|
||
" \"\"\"\n",
|
||
" TODO: Implement the complete pipeline\n",
|
||
" - Fetch all data\n",
|
||
" - Validate each dataset\n",
|
||
" - Generate summary report\n",
|
||
" - Return results\n",
|
||
" \"\"\"\n",
|
||
" pass\n",
|
||
"\n",
|
||
"# Your implementation here:\n",
|
||
"\n",
|
||
"\n",
|
||
"# Test your pipeline\n",
|
||
"# pipeline = BinanceDataPipeline(\n",
|
||
"# symbols=['BTC/USDT', 'ETH/USDT', 'BNB/USDT', 'ADA/USDT'],\n",
|
||
"# timeframe='1h',\n",
|
||
"# use_credentials=True\n",
|
||
"# )\n",
|
||
"# results = pipeline.run_pipeline(limit=100)\n",
|
||
"# print(results)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Summary\n",
|
||
"\n",
|
||
"In this notebook, you learned:\n",
|
||
"\n",
|
||
"1. **CCXT Library Setup**: How to install and configure CCXT for Binance\n",
|
||
"2. **Secure API Integration**: Proper use of environment variables and dotenv for Binance API keys\n",
|
||
"3. **Binance Connectivity**: Establishing authenticated connections with proper error handling\n",
|
||
"4. **Data Fetching**: Retrieving ticker, OHLCV, and order book data\n",
|
||
"5. **Rate Limiting**: Implementing rate limits to avoid API restrictions\n",
|
||
"6. **Error Management**: Handling network errors, rate limits, and exchange downtime\n",
|
||
"7. **Reconnection Logic**: Building robust connections that auto-recover\n",
|
||
"8. **Market Analysis**: Advanced data processing and technical analysis\n",
|
||
"\n",
|
||
"### Next Steps:\n",
|
||
"- Complete the data pipeline exercise\n",
|
||
"- Move to Session 3.2 for technical indicators\n",
|
||
"- Practice with different timeframes and symbols\n",
|
||
"- Explore authenticated endpoints like trading history and account info\n",
|
||
"- Learn about WebSocket connections for real-time data\n",
|
||
"\n",
|
||
"### Key Takeaways:\n",
|
||
"- Always implement proper error handling for production systems\n",
|
||
"- Rate limiting is crucial for maintaining API access\n",
|
||
"- Connection monitoring prevents silent failures\n",
|
||
"- Secure credential management is essential\n",
|
||
"- Binance offers extensive market data for analysis\n",
|
||
"- CCXT provides a standardized interface for cryptocurrency exchanges\n",
|
||
"\n",
|
||
"### Binance API Resources:\n",
|
||
"- [Binance API Documentation](https://binance-docs.github.io/apidocs/spot/en/)\n",
|
||
"- [CCXT Documentation](https://ccxt.readthedocs.io/en/latest/)\n",
|
||
"- [Binance API Management](https://www.binance.com/en/my/settings/api-management)"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.8.5"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 4
|
||
}
|