Resources | Yaakov Bressler

Resources

Will be providing links here for the hackathon situation:

Datasets (GCP Bucket = `koala-onesie-verona-2023/`):

Be careful when opening these links, they will automatically download a bunch of massive data files…

Key Datasets: ⭑

California Traffic Collision Data from SWITRS (archive.zip)
- Source: Kaggle
CA Weather (weather_CA_2019.csv.gz)
LA Traffic Collision Data (Traffic_Collision_Data_from_2010_to_Present.csv.gz)
- Source: LA Open Data - Traffic Collision Data from 2010 to Present

Other Interesting Datasets:

FTC Complaints (complaints.csv.gz)
CA Traffic Volumes
- Traffic (Traffic_Volumes_AADT.csv)
- Truck (Truck__Volumes_AADT.csv)
- Source:
  - California Open Data Portal (reference)
  - California Department of Transportation (true source)
LA specific:
- LA Parking Violations (Parking_Citations.csv.gz)
  - Source: LA Open Data - Parking Citations
- Vehicle and Pedestrian Stop Data (Vehicle_and_Pedestrian_Stop_Data_2010_to_June_30th__2018.csv.gz)
  - Source: LA Open Data - Vehicle and Pedestrian Stop Data 2010 to June 30th, 2018

Vehicle Deaths:
- U.S. Department of Transportation’s Fatality Analysis Reporting System (FARS).
- Fatality Facts 2020, State by state
- Transportation reporter Tanya Snyder’s tweet about car deaths
- CA Highway Patrol - SWITRS 2019 Annual Report of Fatal and Injury Motor Vehicle Traffic Collisions
Vehicle, Fuel, and Road Conditions:
- Fuel Economy Data (not that helpful tbh…)
- California Department of Transportation - Traffic Manuel

Tools & Resources:

Key Stuff

My Github: https://github.com/ybressler
Mess Around Repo: https://github.com/ybressler/mess-around-duckdb

Helpful commands:

Zip a csv (compression):
```
  gzip -kv9 data.csv
```
Flags:
- k: keep original file
- v9: use “best compression algorithm”
Unzip a csv (decompression):
```
  gunzip -k data.csv.gz
```
Create a new python virtual environment:
```
  python3 -m venv venv
```
Arguments:
- python3 use this executable (in path) to do the following…
- -m: make
- venv a virtual environment
- venv (could be any name), store the virtual environment in “this directory”
The same command works too:
```
   python3 -m venv my_virtual_environment_name
```
Activate a virtual environment:
```
  source venv/bin/activate
```
Where venv/ is the directory of your virtual environment
Install dependencies (from requirements.txt):
```
  pip install -r requirements.txt
```
Store dependencies into a requirements.txt:
```
  pip freeze > requirements.txt
```

Helpful Images

Tanya Snyder's tweet about car deaths

Tanya Snyder's tweet about car deaths

Comparing DB engines (we like DuckDB)

Comparing DB engines