while (true) {

Jump to latest Follow Reply
K

koala

Ars Tribunus Angusticlavius
7,579
  • #22,041
    Oh, actually dependencies are not a big issue. One good thing about the Android build system is that it bundles most of the stuff it requires, and what's not bundled is quite easy to install.

    The complexity comes more from a sizable amount of build scripts that have a good amount of complexity and that must fit together. Another parts come from us not knowing well enough the Android build system, and doing extra work to perform some building outside that.

    Kubernetes could help slightly with running the development environment- because the running environment has a few moving pieces and k8s is good for orchestrating that kind of thing, but I feel it's not really worth it.

    ...

    It's also a product of a full build taking 45 minutes to run, so experimentation is a bit slow (we don't need to do full builds often, but when you're messing with the build system, you want to test full builds).

    ...

    Surprisingly, this team has an obsession with doing as much as possible on Rust. And I thought that was a terrible idea for the build scripts, but... the ability to catch most errors at compile time when working on the build scripts has really been worthwhile. I was already a convert of replacing shell scripts with Python scripts, but I'm slowly becoming convinced that using a statically-typed language is likely worthwhile. Also in build scripts you really don't hit the difficult bits of Rust, so actually you get the pleasant bits, but not much of the painful bits.
     
    • Like
    Reactions: dio82, MilleniX and Klockwerk
    Apteris

    Apteris

    Ars Tribunus Angusticlavius
    8,938
    Subscriptor
  • #22,042
    koala said:
    The complexity comes more from a sizable amount of build scripts that have a good amount of complexity and that must fit together.
    Click to expand...
    We have this at work too, including the 45-minute run time for the full CI pipeline (with tests).

    The build system is a software project. As with most other software projects, you iterate, get to know it better, and can then make it smaller and more efficient.

    I do recommend that you put all parts of the build system in the same source control repository, deduplicate what you can, and make the process as modular as is reasonable.

    MilleniX recommended using Infrastructure as Code, which generally I'm in favour of, but I don't know or understand your environment. You mentioned GitHub Actions, so I guess at least part of your CI workflows are executed on "GitHub's" servers (I almost wrote "cloud-native", ick). But then you also mentioned "bare metal beefy servers", which... you know AWS will lease you machines with 64 vCPUs and 256 GiB of RAM, right? If you're building on bare metal [1] as some form of cost optimisation, fair enough, but it doesn't sound to me like a use case that really needs it.

    In any event, whatever infrastructure you're running your builds on, it needs to be reliable and well-managed. So yes, IaC if you can swing it.

    [1] "Bare metal" is, of course, not the thing as "on premises", nor is it the same thing as "not being offered by AWS or another major cloud provider". But I got the impression it meant "in our own data center", in this context.
     
    M

    Mark086

    Ars Tribunus Angusticlavius
    10,595
  • #22,043
    I created a restaurant menu for myself a couple years ago; keeping the prices updated is annoying, particularly recently as they've updated the prices a couple times in the past several months.

    I decided to try an experiment with chatgpt; while I couldn't use the standard chat interface I did use the backend, fed it photos of the menu and told it to update a block of json (the entire menu), to fix prices and descriptions.

    It did it, and even added a section I hadn't bothered to include. (The drink menu, orange juice, Pepsi, etc).

    Impressed.

    Code: 
    System: update assistant to update 
restaurant menu items and prices.
 Don't remove items from the menu, 
we will assume the menu only has price 
and description changes, is may not 
include everything from the original menu.

Prompt: use these images to update 
prices and descriptions and generate a 
new json file:
(JSON pasted in)

    Sure, maybe I could have built a specific workflow with ocr, but this was much easier.

    I imagine it wouldn't be too difficult to build a workflow for photos of any menu (within some reasonable semblance of readability).
     
    • Like
    Reactions: svdsinner and waveguide
    K

    koala

    Ars Tribunus Angusticlavius
    7,579
  • #22,044
    Apteris said:
    I do recommend that you put all parts of the build system in the same source control repository, deduplicate what you can, and make the process as modular as is reasonable.
    Click to expand...

    We're actually monorepoing things, which helps with this.

    Apteris said:
    MilleniX recommended using Infrastructure as Code, which generally I'm in favour of, but I don't know or understand your environment. You mentioned GitHub Actions, so I guess at least part of your CI workflows are executed on "GitHub's" servers (I almost wrote "cloud-native", ick). But then you also mentioned "bare metal beefy servers", which... you know AWS will lease you machines with 64 vCPUs and 256 GiB of RAM, right? If you're building on bare metal [1] as some form of cost optimisation, fair enough, but it doesn't sound to me like a use case that really needs it.

    In any event, whatever infrastructure you're running your builds on, it needs to be reliable and well-managed. So yes, IaC if you can swing it.

    [1] "Bare metal" is, of course, not the thing as "on premises", nor is it the same thing as "not being offered by AWS or another major cloud provider". But I got the impression it meant "in our own data center", in this context.
    Click to expand...

    We're using Hetzner, which is significantly cheaper, and the slightly decreased convenience wrt. AWS is absolutely worth it. We run a GitHub Actions Runner on Hetzner, which is a bit of work, but also very worth it.

    So it's bare metal, but not on-premises.

    We actually have some on-premise hardware, but it's because we do hardware stuff and it's more of a necessity.
     
    moosemaimer

    moosemaimer

    Ars Scholae Palatinae
    717
  • #22,045
    Mark086 said:
    I created a restaurant menu for myself a couple years ago; keeping the prices updated is annoying, particularly recently as they've updated the prices a couple times in the past several months.

    I decided to try an experiment with chatgpt; while I couldn't use the standard chat interface I did use the backend, fed it photos of the menu and told it to update a block of json (the entire menu), to fix prices and descriptions.

    It did it, and even added a section I hadn't bothered to include. (The drink menu, orange juice, Pepsi, etc).

    Impressed.

    Code: 
    System: update assistant to update
restaurant menu items and prices.
 Don't remove items from the menu,
we will assume the menu only has price
and description changes, is may not
include everything from the original menu.

Prompt: use these images to update
prices and descriptions and generate a
new json file:
(JSON pasted in)

    Sure, maybe I could have built a specific workflow with ocr, but this was much easier.

    I imagine it wouldn't be too difficult to build a workflow for photos of any menu (within some reasonable semblance of readability).
    Click to expand...
    Now I wonder how reliably it works in the opposite direction: "given this .json and some base images, construct a restaurant menu"
     
    zeotherm

    zeotherm

    Ars Legatus Legionis
    10,615
    Moderator
  • #22,046
    My rage at MongoDB as a user/developer continues unabated. I swear EVERY time I get asked to do a simple upgrade to their libraries they find a completely new (or old... wait for it) way to complete hose things. I have no idea how one shop can be so unbelievably bad at their job.

    Last week I went to update our PyMongo libraries only to find that when you do a paginated read of a large Mongo collection, the new version of the library will FUCKING display the last document read in your log file at the end of a pagination boundary (as a Mongo cursor). Well, I work with highly confidential medical information and guess what, YOU CANNOT LEAK THAT SHIT TO A FUCKING LOG FILE. Who thought that was a good idea/acceptable behavior should never be allowed near a database again.

    Today I am trying to update some Scala/Java drivers they have and well, if you simple bump the version numbers, they have screwed it up so that the JAR cannot be built because they attempt to include (one of their internal dependencies) the same jar twice screwing things up. I have been fucking around with sbt build and merge strategies ALL FUCKING DAY (which is an abject waste of my time). What KILLS me though is THIS ISN'T THE FIRST TIME THEY FUCKED THIS UP! Stackoverflow posts from 2021 show they did a near identical screw up when they released the 2.13.3 driver libraries and they didn't fix them until the 2.18.0 release.

    I swear if I had any opportunity to drop them in out current software (they are gone in the next major release) I would do it in a second. I cannot get over how terrible they are at developing stable*, cromulent software

    * Of course NONE of this touches on the complete shit performance regressions they keep introducing. What ass clowns
     
    invertedpanda

    invertedpanda

    Ars Tribunus Militum
    1,844
    Subscriptor
  • #22,047
    Jesus.

    I keep thinking I should build something production-level with Mongo just to prove that I can, but then I'm like.. "Why".

    In every situation where I've considered building something that'd need a DB, there has been little to no reason to not just use plain ole' SQL.
     
    • Like
    Reactions: dio82, Neverm1nd, VirtualWolf and 1 other person
    zeotherm

    zeotherm

    Ars Legatus Legionis
    10,615
    Moderator
  • #22,048
    invertedpanda said:
    I keep thinking I should build something production-level with Mongo just to prove that I can, but then I'm like.. "Why".
    Click to expand...
    Yeah, take my advise and don't...

    I don't begrudge the devs who came before me in the earliest days of the startup we're all at. Mongo (warts and all) was the right decision for getting us off the ground. But now 7 (?) years later where we are damn near fully established, that decision hangs around our necks like a millstone :(

    I do not understand the use case for Mongo in any mature software product. I can't see choosing it over traditional database.
     
    zeotherm

    zeotherm

    Ars Legatus Legionis
    10,615
    Moderator
  • #22,051
    Jonathon said:
    I've seen it used sort of well in an application or two that wasn't a good fit for the relational database paradigm... but the best example I have of that probably would've been even better served by a graph database (in the same class as JanusGraph or Neo4j).
    Click to expand...
    I would give anything for a high quality graph database for some of the data we work with but even a nice relational db is a welcome change :)
     
    M

    MilleniX

    Ars Tribunus Angusticlavius
    6,767
    Subscriptor++
  • #22,057
    ShuggyCoUk said:
    My loaner is a 16”. It’s enormous so I’m glad I went for 14”

    It’s an interesting place and I suspect you will be able to work it out based on my thoughts and questions but I think I will keep my approach of never directly naming my employer for now. If only for the mystique 🥸🫥
    Click to expand...
    That's fair. I guess I'll wait until clues start to accumulate.
     
    herko

    herko

    Ars Praefectus
    5,676
    Subscriptor++
  • #22,058
    invertedpanda said:
    Jesus.

    I keep thinking I should build something production-level with Mongo just to prove that I can, but then I'm like.. "Why".

    In every situation where I've considered building something that'd need a DB, there has been little to no reason to not just use plain ole' SQL.
    Click to expand...
    PostgreSQL, PostgreSQL, PostgreSQL. All day, every day. It's fantastic. I've played with Couchbase, looked at Mongo, and it's always the same conclusion: why would I do this?

    There's graph extensions, too.
     
    • Like
    Reactions: dio82, dspariI and invertedpanda
    G

    gregatron5

    Ars Legatus Legionis
    11,245
    Subscriptor++
  • #22,059
    My parents are having a huge gathering on a cruise to celebrate their 50th anniversary. Royal Caribbean's app is just crap. So I thought, maybe I'll see if they're hiring backend engineers?

    They are. Experience required: PHP and MySQL.

    No wonder it's shit. And I don't even want to think about trying to make it better anymore. I'll just suffer.

    This post brought to you by: Postgres. Just use it.
     
    • Like
    Reactions: dio82
    herko

    herko

    Ars Praefectus
    5,676
    Subscriptor++
  • #22,062
    Apteris said:
    Some colleagues at work are taken with DocumentDB. One even expressed enthusiasm for the lack of schemas.

    Am I doomed?
    Click to expand...
    Yes, doomed, unless you drink the lack-of-schema koolaid. There may be reasons to do so. Is the work handling TRULY schema-less data? Web pages? General medical records? Legal documents? If so, maybe they have a case.

    Otherwise, if it leans towards the traditional CRUD application side of the spectrum, run away.
     
    Apteris

    Apteris

    Ars Tribunus Angusticlavius
    8,938
    Subscriptor
  • #22,064
    herko said:
    Is the work handling TRULY schema-less data?
    Click to expand...
    Eh. Without wishing to give away too much, I can say this pertains to the general area of Payments, a domain which turns out to have a good deal of heterogeneity. That is to say, different payment providers do things differently enough that it's worth asking how much schema is desirable.

    Still need to dig into it some more before I can offer an informed opinion.
     
    herko

    herko

    Ars Praefectus
    5,676
    Subscriptor++
  • #22,065
    Apteris said:
    Eh. Without wishing to give away too much, I can say this pertains to the general area of Payments, a domain which turns out to have a good deal of heterogeneity. That is to say, different payment providers do things differently enough that it's worth asking how much schema is desirable.

    Still need to dig into it some more before I can offer an informed opinion.
    Click to expand...
    Fair enough, but the presence of money and numbers (very generally speaking) suggests structured data. At least as a prior.

    You can always add a BLOB or a JSON column to a structured database these days; extracting structure repeatedly from a schema less store is a pain.
     
    • Like
    Reactions: dio82 and Apteris
    Ardax

    Ardax

    Ars Legatus Legionis
    19,076
    Subscriptor
  • #22,066
    Apteris said:
    That is to say, different payment providers do things differently enough that it's worth asking how much schema is desirable.
    Click to expand...
    I think it's one of those areas where you want "just enough schema" to make your own work sane and easy and the original blob of stuff stored wholesale alongside.
     
    • Like
    Reactions: dio82, ShuggyCoUk and Apteris
    H

    hanser

    Ars Legatus Legionis
    41,687
    Subscriptor++
  • #22,067
    I don't know anything about DocumentDB or Mongo, but we use dynamodb, like, everywhere. It's schemaless besides your primary key and optional "sort" key. Our typical pattern is to have a couple of "columns" that contain metadata that's interesting to humans, and then a gzipped blob representing a JSON-encoded object that's the payload / document. The data layer of our applications decompresses it and deserializes it. (We have a lot of GetObject<T> in our data layers.)

    This makes it pretty easy to calculate a bunch of answers to something, write them down in a database that has single-digit ms latency that can go right out the other end of a REST endpoint. You can also upgrade that to microsecond latency if you want to spend more money; at that point, deserialization becomes your bottleneck.

    The advantages are:
    • Not having to do schema migrations
    • Trivial per-customer datastores, each with their own encryption keys. (Usually the same encryption key that encrypts the customer's S3 bucket(s).)
    • IAM permissions that can differ by runtime context.
    • No always-on compute just to serve data
    • No ORMs
    • You can opt into the consistency semantics you prefer
    • Flexible capacity, including ramping up and down based on realtime demand

    For us, it was a great choice. We write down lots of little "report" blobs that contain answers to complex calculations, and we add new "reports" pretty consistently. Ddb is pretty much perfect for this use case. You could duplicate some of this in an RDBMS (just have a column that's VARCHAR(MAX) that you stick compressed content into) but not all of these capabilities can be satisfied without a lot of effort.
     
    herko

    herko

    Ars Praefectus
    5,676
    Subscriptor++
  • #22,068
    hanser said:
    Not having to do schema migrations
    Click to expand...
    You do do them, just in your own code.:judge:

    Unless you plan to never, ever, ever change the document layout/contents.

    I'm not saying it's a bad choice. It sounds great for your use case/implementation. But your code has to look at the contents of the blobs at some point, and there you're dealing with different 'schemas' whether you want to or not.
     
    • Like
    Reactions: technophile
    Lt_Storm

    Lt_Storm

    Ars Praefectus
    16,294
    Subscriptor++
  • #22,073
    Apteris said:
    One even expressed enthusiasm for the lack of schemas.

    Am I doomed?
    Click to expand...
    Really, it is a question of how many hands are in the pie. If there are too many hands and your data system strictly verifies schemata, then it can cause problems, and enthusiasm for lack of schemata makes some sense. Though, then you are really best off with less strict validation, if there is an unexpected field, who cares, is only a problem when an important field is missing or corrupted.
     
    Apteris

    Apteris

    Ars Tribunus Angusticlavius
    8,938
    Subscriptor
  • #22,075
    gregatron5 said:
    Multiple slightly different data definitions in a finance context? I'd bet dollars to pesos someone will recommend XSLT.
    Click to expand...
    I've worked with XSLT, long ago. Really cool language, and great for simple transformations, but IMHO for complex transformations it becomes more difficult to use than a "plain ol'" imperative programming language.

    (Please tell me no JSON analogue of XSLT exists...)
     
    G

    gregatron5

    Ars Legatus Legionis
    11,245
    Subscriptor++
  • #22,078
    Apteris said:
    I've worked with XSLT, long ago. Really cool language, and great for simple transformations, but IMHO for complex transformations it becomes more difficult to use than a "plain ol'" imperative programming language.

    (Please tell me no JSON analogue of XSLT exists...)
    Click to expand...
    I've worked with XSLT at various times in my career. It started right out of the gate at my first programming job when XML and XHTML were still the new hotness. I marshaled SQL results into XML and used XSLT to combine them with XHTML fragments to make XHTML web pages. Since then I've used it on and off. Most recently it was used most recently to massage several similar standards of XML into another internal standard for processing at a scale that's pretty mind boggling.

    It's esoteric and kind of fascinating when you really dig into it. I can't say I miss it, but I'm glad I had the experience of figuring it out and working with it. It's like nothing else I've ever used.
     
    • Like
    Reactions: AdrianS
    H

    hanser

    Ars Legatus Legionis
    41,687
    Subscriptor++
  • #22,080
    Is AutoMapper in C#-land basically like the POCO equivalent of XSLT? I never had to use the latter, and I view the former as an anti pattern.

    Edit, this is what Llama said:

    AutoMapper and XSLT share some similarities, but they serve different purposes and have different design centers.
    Similarities:
    1. Both AutoMapper and XSLT are used for transformation: AutoMapper transforms objects, while XSLT transforms XML documents.
    2. Both provide a way to map source data to a target structure.
    3. Both support conditional logic and manipulation of data during transformation.

    Differences:
    1. Purpose: AutoMapper is primarily used for object-to-object mapping, typically for domain model mapping, while XSLT is designed for transforming XML documents.
    2. Data format: AutoMapper works with .NET objects, while XSLT operates on XML.
    3. Transformation approach: AutoMapper uses a convention-based mapping approach, while XSLT uses a declarative, template-based approach.
    4. Complexity: XSLT is generally more complex and powerful than AutoMapper, due to its ability to handle complex XML transformations and conditional logic.
    In summary, while AutoMapper and XSLT share some conceptual similarities, they are designed for different purposes and have different design centers.
     
    You must log in or register to reply here.
    Jump to latest Follow Reply
    Jump to latest Follow Reply