Replacement for OpenWeatherMap.org?

Jump to latest Follow Reply
stevenkan

stevenkan

Ars Legatus Legionis
15,662
  • #1
    I'm logging the weight of my "science hive" every 10 minutes, and I thought it would be neat to plot the weight gain against the reported temperature, with the temperature being a rough proxy for overall weather, and therefore for the availability of food, since I live in dry-ish Southern California. As you can see, the temperature data are really noisy:

    1714604570970.png

    The excessively granular, 10-minute log is my raw data from openweathermap.org, and I will eventually replace with with something like a 24-hr moving average, as I've done with colony weight, but the data are so noisy that it's going to throw off even the 24 hr average.

    Is there another free weather source that I could use?

    I'm currently collecting the weight from 4 stamp.com postage scales, temp and humidity from openweathermap.org, and temp and humidity from a TemperHum sensor, in a single bash script, where the weather call is of fhe form:

    Code: 
    TheWeather=`curl -s 'api.openweathermap.org/data/2.5/weather?id=5388601&APPID=<MyUniqueAPICode>&units=imperial' | jq -j '.main.temp," ", .main.humidity'`

    which returns the bolded sections of a record like this:

    2024-05-01-13:50:01 271.4 188.8 55.8 149.6 65.62 72 73.92 69

    e.g.

    echo $TheDateTime $TheWeight1 $TheWeight2 $TheWeight3 $TheWeight4 $TheWeather $TheTemperHum >> $OutFile

    so ideally I'd like to replace openweathermap.org with something that I can drop directly into my existing script.

    Any good, free candidates out there?

    Thanks!

    Or is this better suited to the Programmer's Symposium?
     
    Arcturus

    Arcturus

    Ars Tribunus Angusticlavius
    6,623
    Subscriptor
  • #2
    A handful of thoughts...

    If you have local data, do you need internet data for temperature or humidity?

    It looks like the data is "predictably unpredictable"? Could you do input checking and only allow for a "reasonable" temperature rise in 10 minutes, throwing out data points that are obviously wrong, and then smooth from that?

    Do you have an airport nearby? Looks like you can get METAR data through an API: https://aviationweather.gov/data/api/
     
    Jonathon

    Jonathon

    Ars Legatus Legionis
    16,541
    Subscriptor
  • #3
    The National Weather Service has an API; it's the same data that powers their website: https://www.weather.gov/documentation/services-web-api

    There's a single endpoint for current conditions at a station; for example: https://api.weather.gov/stations/KBNA/observations/latest

    You'll need a station ID near you; if you search for your local forecast on weather.gov (box at the top left corner), this will be the four-letter ID in parentheses underneath "Current conditions at". (There's an API for this, too, but that's probably a bit complicated for what you want to do here.)
     
    • Like
    Reactions: continuum and stevenkan
    stevenkan

    stevenkan

    Ars Legatus Legionis
    15,662
  • #4
    Thanks! aviationweather.gov works, but doesn't include humidity.

    I found weatherapi.com:

    Code: 
    curl -s 'http://api.weatherapi.com/v1/current.json?key=<MyFreeAPIKey>&q=90274&aqi=no' | jq -j '.location.name,", ",.'current.temp_f," ºF, ",.current.condition.text'
    which returns:

    Palos Verdes Peninsula, 62.6 ºF, Partly cloudy

    and even:

    Code: 
    curl -s 'http://api.weatherapi.com/v1/current.json?key=<MyFreeAPIKey>&q=90274&aqi=no' | jq -j '.current.condition.icon'
    which returns:

    Code: 
    //cdn.weatherapi.com/weather/64x64/day/116.png
    which returns:
    116.png

    I think it remains free after 14 days. We'll find out.
     
    stevenkan

    stevenkan

    Ars Legatus Legionis
    15,662
  • #5
    Jonathon said:
    The National Weather Service has an API; it's the same data that powers their website: https://www.weather.gov/documentation/services-web-api

    There's a single endpoint for current conditions at a station; for example: https://api.weather.gov/stations/KBNA/observations/latest

    You'll need a station ID near you; if you search for your local forecast on weather.gov (box at the top left corner), this will be the four-letter ID in parentheses underneath "Current conditions at". (There's an API for this, too, but that's probably a bit complicated for what you want to do here.)
    Click to expand...
    Thanks! I will try this, too. I was hoping to find (or to be directed towards) something like this, as I'd expected to find this available from .gov. Especially since I'm a tiny cog in their efforts to build the next generation of weather satellites.
     
    stevenkan

    stevenkan

    Ars Legatus Legionis
    15,662
  • #6
    Jonathon said:
    There's a single endpoint for current conditions at a station; for example: https://api.weather.gov/stations/KBNA/observations/latest
    Click to expand...
    Ok! If I put in my lat/lon:


    it returns some stuff that includes a link to a forecast for a polygon that includes me:


    and that returns JSON from which I can extract data for my graph:

    Code: 
    curl -s  https://api.weather.gov/gridpoints/LOX/149,33/forecast | jq -j '.properties.periods[0].temperature," ", .properties.periods[0].relativeHumidity.value'

    61 88

    or text for my video overlay:

    Code: 
    curl -s  https://api.weather.gov/gridpoints/LOX/149,33/forecast | jq -j '.properties.periods[0].temperature,"ºF, ",.properties.periods[0].relativeHumidity.value,"%, ",.properties.periods[0].shortForecast'

    61ºF, 88%, Slight Chance Light Rain
     
    • Like
    Reactions: Baenwort and continuum
    stevenkan

    stevenkan

    Ars Legatus Legionis
    15,662
  • #8
    Jonathon said:
    Yep. Although that's forecast data, not actual current conditions, if that matters.
    Click to expand...
    Yeah, I was reading this discussion on GitHub about the API:

    The NWS API does not provide that kind of data. (At least as of the date of this post.)

    By definition, there is no "reported current temperature" where "there is no station nearby." If a website is giving you a "current temperature" at your location and you don't have a reporting station there, one of 2 things is happening:
    • The site is producing a synthetic value by interpolating between reporting stations; or
    • The site is referencing a forecast model and giving you the temperature the model thinks it should be at or near your location.
    In the case of the NWS API, it only gives you current temperatures for reporting stations the NWS tracks, which are usually major installations like airports.
    Click to expand...
    I'm not very climatologically close to a reporting station, so I figured NWS's closest forecast for my lat/lon would be more accurate than the actuals over there. I'm in a bit of a mild canyon on a north facing slope. The Torrance airport (KTOA) is only 3 crow miles away, but 800' lower in elevation in the middle of a pretty flat basin.

    If that person on GitHub is correct, every current condition for my location is based on a model.

    I'll probably log NWS vs. weatherapi.com vs. openweathermap.com for a few days and how they compare.
     
    Jonathon

    Jonathon

    Ars Legatus Legionis
    16,541
    Subscriptor
  • #9
    There's no "if" involved-- no weather service has the magic psychic power to know what temperature it is at your exact location unless you put a thermometer there (this is the premise behind Weather Underground, OpenWeatherMap, and other similar sites).

    If a reliable temperature reading at your specific location is what you need, there's really no substitute for putting your own temperature sensor outside. If you're lucky and the microclimate created by your mild canyon and slope isn't too small or too weird for the models to catch, you might get close with the models, but it won't be exact.

    (If that gray line in your first post is coming from a sensor that you're running, something is broken-- temperature sensor data shouldn't look like that, even if poorly placed.)
     
    A

    AverageDutchGuy

    Ars Scholae Palatinae
    681
  • #11
    Looks like OpenWeatherMap.org is consolidating/reporting data from 2 different sensors somewhere in the vicinity of your actual location which don't agree/mismatch on the measured temperature (one more shaded than the other?). I think the only good option you have to get actual local data is to install your own temperature and humidity sensors. Shouldn't be too hard to buy a cheapish weather station and hack it's comms protocol (or buy one that already had it's comms protocol hacked).
     
    stevenkan

    stevenkan

    Ars Legatus Legionis
    15,662
  • #12
    Interesting. Right around the same time that I started looking into this, it looks like OpenWeatherMap started fixing the problem. Those "hairs" start to go away around 5/9 - 5/10:
    1716351308521.png

    I wanted to clean up the dataset, so I added a "correction" column where I reject any 10-minute change in temp > 3 ºF:

    1716351353369.png

    plus some other fudge factors to clean up missing data, e.g. 4/8 - 4/9, when I kicked my Pi off of the internet for a few hours.

    Then I added a 24-hr average:

    1716351412191.png

    and then scaled it so it looks better wrt to the hive data (the prev. 3 charts are scaled to the same Y axis values, to better illustrate the effect of data cleanup and smoothing):

    1716351450932.png

    The temperature's been remarkably constant for the last month or so. We'll probably start warming up in mid-June.
     
    • Like
    Reactions: Baenwort
    You must log in or register to reply here.
    Jump to latest Follow Reply
    Jump to latest Follow Reply