Is distributed compute dying?

  • Thread starter Deleted member 32907
  • Start date
Jump to latest Follow Reply
D

Deleted member 32907

Guest
  • #1
    My office heaters are getting disturbingly far down the list of "things they enjoy doing," while scrounging for work that isn't synthetic math makework.

    I try to throw my cycles at things that are "actual, real-world problems" in the physical space, partly because I've got a rather comically oversized solar array on my office that powers it, and I'd rather put the surplus to something useful (though I can't pull nearly enough to actually tax the array, it just idles along most days - I need another inverter off the battery bank and an outdoor case for servers).

    Climateprediction.net used to have endless work, and that well has been rather dry lately - with any trickles into it getting slurped up in a hurry by the handful of users with a lot of machines (of which I'm one). They're a fiddly pain in the rear to work with - they still ship 32-bit tasks for Linux and MacOS, which means you need 32-bit libraries for Linux, and for MacOS, most of the actual work is being done with Mojave VMs on Linux or Windows hosts, as MacOS hasn't supported 32-bit tasks for... well, Mojave was the last. As a general rule of thumb, the tasks don't suspend/resume successfully, so if you do anything but "let the task run straight through from start to completion," they're likely to crash on you. At least system suspend doesn't interfere... but neither do they run on "most systems" without problems.

    World Community Grid is still down. They've been down for the move, and... are still down, might come back at some point, but some of the errors they've talked about in their updates don't give me any great confidence in the skills of the people managing the infrastructure right now.

    Rosetta has lots of crashing tasks, and generally no work anyway.

    F@H prefers GPUs, which are still unobtanium in any reasonable performance... all my rigs are CPU based right now, because my GPUs are getting used by people for actual GPU work anymore. And the best thing I have is an old crufty 980.

    Eintein@Home is filling my queues right now, but it's still pretty far down the list of things I'd rather be running.

    And then there's the math stuff, but... I'm just not that interested in throwing very real energy down that rabbithole.

    And this forum is quite dead.

    It seems like it's just a bunch of the old legacy grid compute types running stuff, and I'm assuming the ease of cloud compute and grants has taken over a lot of the compute needs.

    Anything interesting I'm missing lately?
     
    B

    Burned

    Ars Tribunus Militum
    2,530
  • #3
    I don't think you're missing anything interesting, but I also don't really know where to look for current general DC conversation. [H}ardOCP is moderately more active than here, although that sure wouldn't take much. Personally, I'm just plodding along doing Folding, hoping all this work might actually lead to something useful. Points systems definitely skew towards GPU work, but I think that is just the nature of the data analysis DC does. I do kinda miss the old days.
     
    J

    JimboPalmer

    Ars Tribunus Angusticlavius
    9,402
    Subscriptor
  • #4
    https://folding.extremeoverclocking.com ... ary.php?s=

    (Click active users, monthly)

    For a shining moment, twenty times as many people tried Folding@Home in hopes of cureing COVID. Then they got bored, could not afford the electric bill, or were persuaded that COVID was a left wing conspiracy that could be cured with bleach.

    Twice as many people are still folding, however.
     
    Last edited:
    • Like
    Reactions: babyachilles
    B

    Burned

    Ars Tribunus Militum
    2,530
  • #5
    F@H must have gotten some mainstream media pimpage for that to happen (I don't remember specifically what drove users to the site, but do know they had growing pains with the huge influx of users). The rest of the DC metaverse looks about like Arecibo. I guess there are a few healthy-ish BOINC based projects, but there is not a whole lot of discussion that I can find. Stephen Brooks's DC vault site is dead (I guess at least "we" won by his reckoning) and he closed up his project. Free-DC hasn't been active in years.

    I think what happened was Seti@Home got widespread mass media coverage and captured the attention of a whole lot of people. Most got bored, some drifted off to other projects, and SETI couldn't or wouldn't drive the hype train to the next level. Scoring systems changed and the average Joe determined he wasn't going to compete or make a difference and dropped by the wayside. So we're left with just a few hobbyists/enthusiasts, crypto mining side gigs, and the occasional corporate/academic pharm brought to bear for some short period of time. DC doesn't get a lot of (well, really any) editorial coverage any more from the "usual suspects" sites, so its just off the radar for most folks.
     
    D

    DrBoar

    Wise, Aged Ars Veteran
    114
  • #8
    World Community Grid is givning me Rosetta units, so the backend is working even if their frontpage does not :)
    I seem to have goten? new units for close to a week, So I can both receive and send units.

    I think part of the attraction of DC in the 90s was that this was the time of CRTs and screen savers (flying toasters anyone?) and having SETI as a screen saver gave you science/geek cred. (The problem I had was with a 60 MHz power PC turning on the screen saver mode slowed down the number crunching by 5-10 times as most of the CPU was needed to process the screen saver as such.
     
    D

    Deleted member 32907

    Guest
  • #9
    Are you sure?

    I'm still getting maintenance shutdown on WCG.

    Code: 
    02-Jun-2022 11:41:14 [World Community Grid] work fetch resumed by user
02-Jun-2022 11:41:15 [World Community Grid] Sending scheduler request: Requested by project.
02-Jun-2022 11:41:15 [World Community Grid] Requesting new tasks for CPU
02-Jun-2022 11:41:17 [World Community Grid] Scheduler request completed: got 0 new tasks
02-Jun-2022 11:41:17 [World Community Grid] Project is temporarily shut down for maintenance
02-Jun-2022 11:41:17 [World Community Grid] Project requested delay of 86400 seconds

    https://www.worldcommunitygrid.org/server_status.php reports:

    Code: 
    <error>
<error_num>-183</error_num>
<error_string>Project is temporarily down</error_string>
</error>
     
    D

    DrBoar

    Wise, Aged Ars Veteran
    114
  • #10
    Up and running as of now

    Ska-rmavbild-2022-06-02-kl-20-18-58.png
     
    B

    Burned

    Ars Tribunus Militum
    2,530
  • #12
    World Community Grid was IBM's foray into corporate sponsorship of volunteer DC back in the heyday. I think it was both a warm and fuzzy PR move for the company, and an attempt to research actually doing distributed computing for corporate purposes. The latter never came to anything and the underlying engine was migrated to BOINC. I'm not sure if it is now just another BOINC front end, or if it does independent work. Since I've been in IBM's sphere of influence as either a customer or employee, I have probably seen more references to it than the average person.
     
    A

    Acibant

    Seniorius Lurkius
    20
    Subscriptor++
  • #13
    Syonyk said:
    I've no idea why your client says World Community Grid while it's running Rosetta tasks...
    Click to expand...
    They are running a "branded" BOINC client that replaces references to BOINC with World Community Grid. Which can be confusing when you then connect that client to projects that aren't WCG. It was also a sign of IBM losing interest in maintaining things that the configuration was set to also point to an old version of the custom client that was not getting updated.

    I share Syonyk's dislike of mathematics projects. If WCG does not come back there is SiDock@home for biomedical work. Unfortunately it is a Russian-based project so that makes it fraught given the current geopolitics. Even if WCG comes back they have really undermined confidence in missing many deadlines and making excuses that don't really wash. Saying you will launch on a particular date (which happened to be close to a month after the original launch date) and then the day before saying the production environment is being tested makes one wonder. And almost a month has passed after that. WCG was my sole project for a year and with it still down and a poor selection of others to choose from, I would agree with the prospect of distributed computing dying.
     
    D

    Deleted member 32907

    Guest
  • #14
    Acibant said:
    Even if WCG comes back they have really undermined confidence in missing many deadlines and making excuses that don't really wash. Saying you will launch on a particular date (which happened to be close to a month after the original launch date) and then the day before saying the production environment is being tested makes one wonder. And almost a month has passed after that. WCG was my sole project for a year and with it still down and a poor selection of others to choose from, I would agree with the prospect of distributed computing dying.
    Click to expand...

    Seriously. Last update was June 3. It's now June 15, and I'm still getting "Project down for maintenance, try again in a day..." responses.

    At least there's a bit of CPDN work for the moment. And I've got my boxes set up to run CPU Rosetta tasks if I can't get anything off CPDN - there are a lot of hungry Linux boxes, about 2000-2500 tasks will disappear in the first day or two into the work queues of the boxes set up for deep queues (mine included). The work will be gone in about a week, though I've got enough tasks to keep my machines chewing for a month or so (depending on what runs overnight).
     
    B

    Burned

    Ars Tribunus Militum
    2,530
  • #15
    Syonyk said:
    Acibant said:
    Even if WCG comes back they have really undermined confidence in missing many deadlines and making excuses that don't really wash. Saying you will launch on a particular date (which happened to be close to a month after the original launch date) and then the day before saying the production environment is being tested makes one wonder. And almost a month has passed after that. WCG was my sole project for a year and with it still down and a poor selection of others to choose from, I would agree with the prospect of distributed computing dying.
    Click to expand...

    Seriously. Last update was June 3. It's now June 15, and I'm still getting "Project down for maintenance, try again in a day..." responses.

    At least there's a bit of CPDN work for the moment. And I've got my boxes set up to run CPU Rosetta tasks if I can't get anything off CPDN - there are a lot of hungry Linux boxes, about 2000-2500 tasks will disappear in the first day or two into the work queues of the boxes set up for deep queues (mine included). The work will be gone in about a week, though I've got enough tasks to keep my machines chewing for a month or so (depending on what runs overnight).
    Click to expand...

    The general dynamic in IBM over the last few years is that far less people are doing far more work. When a person responsible for something (particularly if it is not revenue generating) goes on vacation, or more likely was let go, the work will not be backfilled. Department mates have enough of their own work and its pretty likely were not cross trained and even less likely there was much documentation created. At that point, the mission (whatever it was) is just going to fall by the wayside. I would not be surprised in the least if WCG went on permanent hiatus.
     
    A

    Acibant

    Seniorius Lurkius
    20
    Subscriptor++
  • #16
    I came across this essay by David P. Anderson titled "BOINC in Retrospect" that seems quite relevant to this thread's subject. He leads the BOINC project so he has first-hand perspective on the situation from a high level. I'd say to pay particular attention to section 18 on the decline of BOINC, 21 with analysis, and 22 with the conclusion. Basically, he says that BOINC never fully achieved his vision and he is disappointed and worn down.
     
    B

    Burned

    Ars Tribunus Militum
    2,530
  • #17
    Acibant said:
    I came across this essay by David P. Anderson titled "BOINC in Retrospect" that seems quite relevant to this thread's subject. He leads the BOINC project so he has first-hand perspective on the situation from a high level. I'd say to pay particular attention to section 18 on the decline of BOINC, 21 with analysis, and 22 with the conclusion. Basically, he says that BOINC never fully achieved his vision and he is disappointed and worn down.
    Click to expand...

    Interesting analysis. I think he's right when he says that researchers don't want to create the project infrastructure with no guarantee that citizen scientists will run it. I think the researchers are generally wrong about that, but ok. My question to him would be then why is Folding at Home so successful? New researchers and projects come on board all the time. I think its because the science itself is significant. There's a steady stream of publications in peer reviewed scientific journals. Researching Covid, or cancer or Alzheimer's seems a lot more consequential than doing an exhaustive search math problem. So I think that people once on board stick to it.
     
    • Like
    Reactions: .劉煒
    timezon3

    timezon3

    Ars Scholae Palatinae
    1,477
    Subscriptor
  • #18
    Burned said:
    Syonyk said:
    Acibant said:
    Even if WCG comes back they have really undermined confidence in missing many deadlines and making excuses that don't really wash. Saying you will launch on a particular date (which happened to be close to a month after the original launch date) and then the day before saying the production environment is being tested makes one wonder. And almost a month has passed after that. WCG was my sole project for a year and with it still down and a poor selection of others to choose from, I would agree with the prospect of distributed computing dying.
    Click to expand...

    Seriously. Last update was June 3. It's now June 15, and I'm still getting "Project down for maintenance, try again in a day..." responses.

    At least there's a bit of CPDN work for the moment. And I've got my boxes set up to run CPU Rosetta tasks if I can't get anything off CPDN - there are a lot of hungry Linux boxes, about 2000-2500 tasks will disappear in the first day or two into the work queues of the boxes set up for deep queues (mine included). The work will be gone in about a week, though I've got enough tasks to keep my machines chewing for a month or so (depending on what runs overnight).
    Click to expand...

    The general dynamic in IBM over the last few years is that far less people are doing far more work. When a person responsible for something (particularly if it is not revenue generating) goes on vacation, or more likely was let go, the work will not be backfilled. Department mates have enough of their own work and its pretty likely were not cross trained and even less likely there was much documentation created. At that point, the mission (whatever it was) is just going to fall by the wayside. I would not be surprised in the least if WCG went on permanent hiatus.
    Click to expand...

    Tale as old as time. That's not just the past few years, that's the past few decades (IBM employee from 2001-2014).
     
    D

    Deleted member 32907

    Guest
  • #20
    Current status:

    WCG is down, again, coming up on a month, and hasn't had reliable work for a while now.

    ClimatePrediction.net (CPDN) spent the new years with a server failure on the upload box, is out of WUs, and is currently... broken? Right now, https://www.cpdn.org/ throws a cert error and points to www.climateprediction.net, eventually, without any of the actual BOINC stuff. Not that it matters, there's no work.

    Rosetta@Home had an awful lot of work, and is... basically out, again. 9.5M WUs have been chewed up in a hurry.

    I suppose I can look at the LHC stuff, but that's mostly VBox work, and I'm rather short on RAM on my compute nodes.

    Einstein@Home has work for a while, if you care about neutron stars.

    I suppose I can point the CPUs to F@H. Even though they hate modern Linux boxes and you're back to configuring the clients by hand, because the control app won't install on anything recent.

    What else is doing anything actually practical?
     
    B

    Burned

    Ars Tribunus Militum
    2,530
  • #21
    Paulie

    Paulie

    Ars Tribunus Militum
    1,514
    Subscriptor++
  • #23
    Team Prime Rib is still going, I just updated the thread. Part of the problem with GIMPS is the numbers being searched for are quite large, so requiring large working sets and hardware, and so take a while.

    On the topic of Distributed Computing, I think the peak was ARScoin to be honest. After that experiment people seem to turn towards mining coins instead of projects.
     
    You must log in or register to reply here.
    Jump to latest Follow Reply
    Jump to latest Follow Reply