twitter-monitor/README.md

# Twitter Monitor

A Rust-based Twitter monitoring service that streams tweets from your home timeline to a web frontend. Supports both cURL-based authentication and Twitter API authentication.

## Features

- Real-time home timeline monitoring
- WebSocket-based streaming
- Detailed response logging to files
- Support for both cURL and API authentication
- Clean, responsive web interface
- Configurable polling intervals with countdown timer
- Robust error handling and retries

## How It Works

The application consists of three main components:

1. **Twitter Client**: Handles authentication and API requests to fetch tweets
2. **WebSocket Server**: Streams tweets to connected clients in real-time
3. **Frontend**: Displays tweets and provides a user interface

## Authentication Methods

### cURL-based Authentication (Recommended)

This method captures an authenticated browser session to access Twitter without API keys:

1. **Get cURL Command**:
   ```bash
   # Using Chrome browser:
   1. Go to x.com (Twitter)
   2. Log in to your account
   3. Navigate to your home timeline (click "For you" or "Following")
   4. Press F12 to open DevTools
   5. Go to the Network tab
   6. Find the "HomeLatestTimeline" request
   7. Right-click on the request
   8. Select "Copy" > "Copy as cURL (bash)"
   ```

2. **Save cURL Command**:
   ```bash
   # Create a file in the config directory
   mkdir -p config
   touch config/curl.txt
   
   # Paste the copied cURL command into curl.txt
   ```

### Twitter API Authentication (Alternative)

For official API access (Not fully implemented yet):

1. **Create Twitter Developer Account** at https://developer.twitter.com/
2. **Get API Keys** from your Twitter Developer Portal
3. **Configure API Keys** in your configuration file

## Installation

1. **Clone the Repository**:
   ```bash
   git clone https://github.com/yourusername/twitter-monitor
   cd twitter-monitor
   ```

2. **Build the Project**:
   ```bash
   cargo build --release
   ```

3. **Configure the Project**:
   Create a configuration file in the `config` directory:

   ```yaml
   # config/session.yaml
   twitter:
     # Path to your curl command file
     curl_file: "config/curl.txt"
     # Or use a bearer token (coming soon)
     # bearer_token: "your_bearer_token"
   
   monitoring:
     # Polling interval in seconds
     polling_interval: 60
     # Maximum tweets to fetch per request
     max_tweets: 10
   
   server:
     # WebSocket server host
     host: "127.0.0.1"
     # WebSocket server port
     port: 8080
   ```

## Running the Project

Start the server with:

```bash
# Development mode
cargo run -- --config config/session.yaml

# Production mode
cargo run --release -- --config config/session.yaml
```

The application will:
1. Parse the cURL command to extract authentication details
2. Start monitoring your Twitter home timeline
3. Start a WebSocket server to stream tweets
4. Save all Twitter API responses to files for debugging

## Project Architecture

### Core Components

1. **TwitterClient** (`src/twitter/client.rs`)
   - Responsible for Twitter API interactions
   - Handles both cURL-based and API token authentication
   - Implements tweet parsing and monitoring logic
   - Includes robust error handling and retries

2. **CurlAuth** (`src/twitter/curl.rs`)
   - Parses cURL commands to extract authentication data
   - Extracts URL, headers, cookies, and request data
   - Provides logging of parsed components

3. **Server** (`src/server.rs`)
   - Manages WebSocket connections
   - Broadcasts tweets to connected clients
   - Implements client heartbeat mechanisms

4. **Main** (`src/main.rs`)
   - Coordinates application components
   - Loads configuration and initializes services
   - Sets up monitoring and broadcasting

### Technical Challenges Solved

1. **Twitter API Authentication**
   - Implemented parsing of cURL commands to extract session cookies
   - Added required feature flags to match Twitter's GraphQL API requirements
   - Handled authentication errors with clear messaging

2. **Robust Data Handling**
   - Implemented safe UTF-8 text handling to avoid panics
   - Added detailed response logging to files for debugging
   - Carefully parsed nested JSON structures

3. **Real-time Communication**
   - Built WebSocket server with proper connection handling
   - Implemented heartbeat mechanism to detect disconnected clients
   - Added broadcast functionality with error handling

4. **User Experience**
   - Added countdown timer between scans
   - Implemented clean logging for better readability
   - Provided detailed status updates during operation

## Troubleshooting

### cURL Authentication Issues

1. **Session Expired**:
   - Twitter sessions expire after some time
   - Generate a new cURL command and update your `curl.txt` file
   - Restart the application

2. **400 Bad Request Errors**:
   - Twitter's API requirements change frequently
   - The application automatically adds required feature flags
   - Response details are saved to `twitter_response_*.txt` files for debugging

3. **Parse Errors**:
   - Twitter's response format may change
   - Check the saved response files
   - Update the parsing logic if necessary

## Configuration Options

```yaml
twitter:
  # Path to your curl command file
  curl_file: "config/curl.txt"
  # Or use a bearer token (coming soon)
  # bearer_token: "your_bearer_token"

monitoring:
  # Polling interval in seconds
  polling_interval: 60
  # Maximum tweets to fetch per request
  max_tweets: 10

server:
  # WebSocket server host
  host: "127.0.0.1"
  # WebSocket server port
  port: 8080
```

## License

This project is licensed under the MIT License - see the LICENSE file for details.