The .NET Thread: For all things .NET
- Thread starter zeotherm
- Start date
technophile
Ars Legatus Legionis
Code:
var threadObservable = this.GetUpdateObservable();
threadObservable.Subscribe(_ => this.MarkUpdated(), e => this.Moonshark(), () => {});Anyone have suggestions on a good way to manage web.configs for different environments? (preferably some open source tool).
Right now we are using TFS vNext as our build orchestration engine, a private nuget feed, chocolatey for package installation - but managing N number of config files for an application has been a chore. Ideally I would like to be able to align a particular branch, with a particular deployment location, and assign it configuration files as necessary.
Ideas?
Right now we are using TFS vNext as our build orchestration engine, a private nuget feed, chocolatey for package installation - but managing N number of config files for an application has been a chore. Ideally I would like to be able to align a particular branch, with a particular deployment location, and assign it configuration files as necessary.
Ideas?
What sort of stuff are you putting in your web.config files? Can they be environment variables instead?
(Most of my web.config files are pretty simple, but I understand they can be very complex.)
(Most of my web.config files are pretty simple, but I understand they can be very complex.)
I have a similar problem, and the real issue is we use a 3rd party library that reads from web.config, so we can't use an environment variable.
We are going to be porting our libraries to a more open internal hosted github solution, so we want to keep things like shared encryption secret keys, production db access, stuff like that out of source control.
Octopus deploy honestly looks like the best bet. But the license cost means I would need to go with another Intel CI tool rather than with TFS vnext that honestly has worked pretty great for us.
Octopus deploy honestly looks like the best bet. But the license cost means I would need to go with another Intel CI tool rather than with TFS vnext that honestly has worked pretty great for us.
Can you put the various environment keys into their own .config file, and then include that file in the main web.config? Then you can just add in the correct .config in your build/install scripts.
That's what we are doing now with powershell (rather web.config references a /config/appsettings.config file). Swap them out as part of the build. Environment config files themselves sit in a folder on the actual machine
/configs/*.We are thinking about having a locked down git repository for storing configs, that only the build manager and the build accounts have access to. Build process will checkout the git repo, pull, and copy the appropriate configs over to the correct environment.
That's what we are doing now with powershell (rather web.config references a /config/appsettings.config file). Swap them out as part of the build. Environment config files themselves sit in a folder on the actual machine
We are thinking about having a locked down git repository for storing configs, that only the build manager and the build accounts have access to. Build process will checkout the git repo, pull, and copy the appropriate configs over to the correct environment.
That's pretty much what I do for various config files (I'm more operations than dev). It's basically configuration management, which in a good shop, straddles the line between Dev and Ops.
Minor, first-world annoyance: I have VS2017, but I can't use C# 7, because the Microsoft Build Tools haven't been released yet, which means the build server can't handle it. 
We still have parts of our build process that haven't been upgraded to VS2015... I don't expect that we'll be moving to 2017 anytime soon.Minor, first-world annoyance: I have VS2017, but I can't use C# 7, because the Microsoft Build Tools haven't been released yet, which means the build server can't handle it.
Actually, I just checked and we've (finally) updated the thing that was building with VS2012 to VS2015 (a couple weeks ago). Progress!The benefits of having control of my build environment.
Cry me a river, I have a C++ app that has to,be compiled on VS2010! You know all those nice C++11/14 features.... yeah....We still have parts of our build process that haven't been upgraded to VS2015... I don't expect that we'll be moving to 2017 anytime soon.Minor, first-world annoyance: I have VS2017, but I can't use C# 7, because the Microsoft Build Tools haven't been released yet, which means the build server can't handle it.
I'd like to upgrade to 17 but I have SSIS/SSRS projects in my solution and need to wait a few months until they update SSDT.
Yeah, the fact that VS was ready before, e.g., the Build Tools is, I think, emblematic of the fact that most dark matter developers do everything in Visual Studio, and that continuous delivery pipelines are still a niche thing, unfortunately.
The "report a problem" dialog and workflow is pretty nice. I reported a bug this morning, and you can take a video of it to demonstrate the issue, and include it with the report. The tool also pulls in a memory dump and some other stuff. The attachments aren't public after the fact, but the bug report itself is:
https://developercommunity.visualstudio ... rn-is.html
The problem reporting tool also includes a search feature so you can search (and vote) on issues you encounter while you're in the process of filing a report.
https://developercommunity.visualstudio ... rn-is.html
The problem reporting tool also includes a search feature so you can search (and vote) on issues you encounter while you're in the process of filing a report.
Meh.
I think continuous-integration developers tend to be less early adopters of major new C# versions and VS versions. As long as the build tools come soonish, I don't see it as a problem. Let the VS-only guys find more bugs and stuff.
you could make a wrapper I guess, but C#7 lets you declare variables in out params so you can do
Code:
if(int.TryParse(s,out var i)
f=f+i;Which seems cleaner than a tuple.
Hrm, reading this blog post, I think MS Build Tools aren't A Thing anymore:
https://blogs.msdn.microsoft.com/vcblog ... ild-tools/
Here's the component list:
https://docs.microsoft.com/en-us/visual ... uild-tools
The installer can be invoked via the CLI:
https://docs.microsoft.com/en-us/visual ... ual-studio
I'm using the GUI, but I went into the individual components and basically just selected the msbuild-related stuff. We'll see how that goes...
I did some research this morning, and it's pretty easy to do. I posted a "how to" on StackOverflow this morning. As of an hour or three ago, we have C# 7 on our build server(s).
I'm allergic to apples but I like oranges (and would like to follow the discussion)
So I just want to comment on all the people fighting me (in the while(true) thread) about how Async/Await is the best thing ever and I"m just doing things wrong.
Even after fixing all of our libraries to be async/await compatible (and hoping we don't have to deal with some of our non-async consumers for a while) we started getting deadlocks again. I downloaded the memory dump and fired up windbg, and what do I see? 700 threads all blocked with the same stack trace.
Unless I am grossly misreading things, all of these threads are waiting due to a System.Net.Http.WinHttpResponseStream.Read(). Pulling the code up on Github it shows them spinning up a Task in order to run Async code in a non-async way, pretty much exactly what I was doing. This of course starts deadlocking, which maxes out the thread pool limit of 682 and everything comes to a grinding halt until the dotnet.exe process is killed.
So, this is going to be awesome to try and work around....
Even after fixing all of our libraries to be async/await compatible (and hoping we don't have to deal with some of our non-async consumers for a while) we started getting deadlocks again. I downloaded the memory dump and fired up windbg, and what do I see? 700 threads all blocked with the same stack trace.
Code:
OS Thread Id: 0x2438 (254)
Child SP IP Call Site
2dbbe744 7728081c [GCFrame: 2dbbe744]
2dbbe7e0 7728081c [HelperMethodFrame_1OBJ: 2dbbe7e0] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
2dbbe860 72233253 System.Threading.Monitor.Wait(System.Object, Int32, Boolean)
2dbbe870 7223326c System.Threading.Monitor.Wait(System.Object, Int32)
2dbbe874 722eadf3 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)
2dbbe8c8 722ba6dc System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken)
2dbbe908 722ba5a9 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken)
2dbbe964 722f8ba6 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
2dbbe970 724550c9 System.Runtime.CompilerServices.TaskAwaiter`1[[System.Int32, System.Private.CoreLib]].GetResult()
2dbbe978 6fc85890 System.Net.Http.WinHttpResponseStream.Read(Byte[], Int32, Int32)
2dbbe994 6fc78c80 System.Net.Http.DelegatingStream.Read(Byte[], Int32, Int32)
2dbbe9a4 6fe7547d System.Xml.XmlTextReaderImpl.InitStreamInput(System.Uri, System.String, System.IO.Stream, Byte[], Int32, System.Text.Encoding)
2dbbe9d4 6fe73034 System.Xml.XmlTextReaderImpl.FinishInitStream()
2dbbe9e4 6fe72fed System.Xml.XmlTextReaderImpl..ctor(System.IO.Stream, Byte[], Int32, System.Xml.XmlReaderSettings, System.Uri, System.String, System.Xml.XmlParserContext, Boolean)
2dbbea14 6fe6f8c7 System.Xml.XmlReaderSettings.CreateReader(System.IO.Stream, System.Uri, System.String, System.Xml.XmlParserContext)
2dbbea34 6fe6ebf6 System.Xml.XmlReader.Create(System.IO.Stream, System.Xml.XmlReaderSettings, System.String)
2dbbea48 10909284 Microsoft.WindowsAzure.Storage.StorageExtendedErrorInformation.ReadFromStream(System.IO.Stream)
2dbbea74 10908cb6 Microsoft.WindowsAzure.Storage.Core.Util.Exceptions+d__0.MoveNext()
2dbbead4 1090882c System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Microsoft.WindowsAzure.Storage.Core.Util.Exceptions+d__0, Microsoft.WindowsAzure.Storage]](d__0 ByRef)
2dbbeb08 109087c0 Microsoft.WindowsAzure.Storage.Core.Util.Exceptions.PopulateStorageExceptionFromHttpResponseMessage(System.Net.Http.HttpResponseMessage, Microsoft.WindowsAzure.Storage.RequestResult, System.Func`4)
2dbbeb54 10900dcb Microsoft.WindowsAzure.Storage.Core.Executor.Executor+d__4`1[[System.__Canon, System.Private.CoreLib]].MoveNext()
2dbbee3c 723143d1 System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.InvokeMoveNext(System.Object)
2dbbee44 722770eb System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
2dbbee78 7231442b System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunnerWithContext.RunWithCapturedContext()
2dbbee9c 722ec367 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
2dbbeec8 722bac20 System.Threading.Tasks.Task.FinishContinuations()
2dbbef38 722b972f System.Threading.Tasks.Task.FinishStageThree()
2dbbef44 722ef715 System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].TrySetResult(System.__Canon)
2dbbef54 723c9505 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].SetResult(System.__Canon)
2dbbef70 6fc97be4 System.Net.Http.HttpClient+d__58.MoveNext()
2dbbefc4 723143d1 System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.InvokeMoveNext(System.Object)
2dbbefcc 722770eb System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
2dbbf000 7231442b System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunnerWithContext.RunWithCapturedContext()
2dbbf024 722ec367 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
2dbbf050 722bac20 System.Threading.Tasks.Task.FinishContinuations()
2dbbf0c0 722b972f System.Threading.Tasks.Task.FinishStageThree()
2dbbf0cc 722ef715 System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].TrySetResult(System.__Canon)
2dbbf0dc 72403fcd System.Threading.Tasks.TaskCompletionSource`1[[System.__Canon, System.Private.CoreLib]].TrySetResult(System.__Canon)
2dbbf0ec 6fc9a7b8 System.Net.Http.WinHttpHandler+d__105.MoveNext()
2dbbf1a0 723143d1 System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.InvokeMoveNext(System.Object)
2dbbf1a8 722770eb System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
2dbbf1dc 7231442b System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunnerWithContext.RunWithCapturedContext()
2dbbf200 722ec1ac System.Threading.Tasks.AwaitTaskContinuation.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
2dbbf20c 7231265d System.Threading.ThreadPoolWorkQueue.Dispatch()
2dbbf25c 72404b5a System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
2dbbf46c 72bf309f [DebuggerU2MCatchHandlerFrame: 2dbbf46c]Unless I am grossly misreading things, all of these threads are waiting due to a System.Net.Http.WinHttpResponseStream.Read(). Pulling the code up on Github it shows them spinning up a Task in order to run Async code in a non-async way, pretty much exactly what I was doing. This of course starts deadlocking, which maxes out the thread pool limit of 682 and everything comes to a grinding halt until the dotnet.exe process is killed.
So, this is going to be awesome to try and work around....
Yeah, there's plenty of crap code in the BCL, film at 10 - sorry :/
The varying quality of the .net core code base is a real problem, and has been since the very beginning. That should at least use the ConfigureAwait(false) idiom, since it's from a library. Perhaps they'll accept a pull request?
Edit: push me pull you
The varying quality of the .net core code base is a real problem, and has been since the very beginning. That should at least use the ConfigureAwait(false) idiom, since it's from a library. Perhaps they'll accept a pull request?
Edit: push me pull you
Would adding a `ConfigureAwait(false)` even fix that though? You are still creating a task and blocking the current thread due to non-async wrapping, so if you get enough of those it is still trivial to blow out your Threadpool limits. For context the root of the azure calls are coming from our logging infrastructure where we log a structured log entry to an Azure append blob for each request coming in (containing parameters (even post ones), referrer information, request uri, user information etc....). So from the looks of it if an azure transient error occurs while writing all of these requests could be blocked over this and explode the threadpool (which also explains the random nature of deadlocks happening :-/
There are just so many issues that are causing this together. The wrapping of async code in non-async in the line I mentioned is one, another is that the azure storage library doesn't use an async API for handling errors, and the third is that it still seems like you can read a Stream in a non-async way which is dangerous since you have no idea if that stream implementation has a blocking call in it or not (in this case it looks like it's passing an http based stream throughout which is the cause of the blocking).
Once you get down into the continuations (awaiters) of another task no it wouldn't fix it properly. Once you're on a thread pool it's not okay to block is the basic rule.
do they expose an asynchronous API for the required info? If not could you write one and submit it back to them.
Doing async right (regardl;ess of the underlying technique) almost always requires that async be infectious 'down' the call chain. In your own threads it's fine to block (if you so desire) but once you head into an async chain it needs to efficiently block.
Most of async should *never* involve threads except in so far as coming back onto the original thread in some affinitized model (what you get 'by default' for GUI apps and marshaling back onto the event loop). Most of the time there is not, and should not, be any thread invovled in the asynchronous aspect. Sadly this sort of thing is common in many API's.
They don't and it doesn't look like it will be easy to do. The Azure storage function is trying to read the error information via an XmlReader.Create() call, but there is no XmlReader.CreateAsync() in the API, only a non-async version. So from my understanding I'd have to update the XmlReader class to have an async Create() call, so it calls all internal streams via ReadAsync(), submit a pull request, wait for the next version of CoreFX to be released (and not sure how that impacts all the NetStandard stuff) and then once that is done then upgrade the Azure Storage library to call CreateAsync() instead of Create().
That or write my own xml reading code. Or theoretically I could force the whole http stream to be read into a memory stream asynchronously but I don't know the side effects of that.
Neither of those options seems like trivial amount of work compared to just ripping out all my structured logging code and just revert to unstructured Trace.TraceInformation()/Trace.TraceError() and let IIS/Azure figure out how to to ship those off to blob storage. It's not going to be nearly as easy to pull information out of Kibana as it is now but I guess that's the price we'll have to pay for now.