A continuous integration guide for .NET with GitLab CI

Posted on Wed 24 May 2017 in continuous-integration, gitlab, .net • Tagged with continuous-integration, gitlab, .net

Here, I describe the process of setting up continuous integration (CI) for a C# project that is hosted on GitLab. GitLab is a code sharing platform similar to GitHub but has two distinct advantages:

  • you can create private repositories for free,
  • it includes a free continuous integration tool called GitLab CI that can be run either on GitLab CI servers, or on your own server.

Actually, GitLab even provides their CI servers for free, called Shared Runners, but at the time of this writing only Linux servers are available, so in this guide we are going to use our own Windows server to perform the actual builds and tests on. This means that to follow this guide, you have to have a Windows instance available in order to run a build. For a production use this is ideally a computer that runs 24/7, i.e. a computer on premises or a VPS, but can, for starters, also be your own laptop/pc.

The continuous integration process we will do will be the following:

  1. You commit & push a set of code changes to a GitLab repository.
  2. Your code is immediately automatically pulled from the GitLab repository to the Windows instance.
  3. The C# compiler tries to build the code.
  4. Badges are updated with the status.

In the end, you get a report of everything that happened, together with a badge like this (if the build was successful):

A Success Build Badge

To do this, let's create a test project in Visual Studio. We are going to do the build first and add automatic testing later.

This is the sample code we are going to build:

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("Hello, World!");
    }
}

Create a new repository on GitLab. Next, install GitLab Runner, which is a program that runs on your Windows machine that is connected to the repository and that runs the build. Download the program from here where you will also find more details on how to install it. It's not complicated, but the one thing you'll also need is a token with which you connect Runner with the repository. You find it by opening your repository, and going to Settings -> CI/CD Pipelines. Copy the token and follow the instructions on how to install and connect the runner.

Note: If you only want to build this project on your server, click Disable Shared Runners so GitLab won't try to build this project on their servers.

After this, the next step is to configure how exactly the build should be performed. This is done by creating a .gitlab-ci.yml file in the root of the project directory and specifying how to do the build. Here's a simple example:

build:
    stage: build
    script:
            - C:\Tools\nuget.exe restore
            - cmd.exe /C " "C:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe" /p:Configuration=Release /t:Clean;Build YourSolutionName.sln " && exit %%ERRORLEVEL%%

You have to specify your location of nuget.exe and MSBuild.exe, and the solution name. You may download nuget.exe from here. See this StackOverflow page to find out how to get MSBuild.exe.

After that, every time a code is pushed on the repository, it will be automatically downloaded to the Windows machine that is running GitLab Runner. The build status will be reported under Pipelines.

GitLab Pipelines

Next, you can show the latest build status on the main repository page by copying a Markdown badge code, located under Settings -> CI/CD Pipelines

A Success Build Badge


Which language to use with TWS API (Interactive Brokers API)?

Posted on Sun 26 March 2017 in finance • Tagged with investing, api, interactive brokers, tws

TWS API

Interactive Brokers has an API called TWS API which you can use to programmatically execute trades, poll current/historical market data, get account info, open positions, etc. Official bindings for several programming languages exist, namely: C#, Java, C++, VB, and Python.

All the TWS API functionality is supported in all of those programming languages. Which language should you choose for your trading program?

C

C# is a programming language made by Microsoft, although you can run it on Linux too. It's a statically-typed, which means all variable types are known at the compile time. That means your IDE, in this case probably Visual Studio, knows all the variable types and can help you a lot when writing code.

Since C# is primarily a language that runs on Windows, you'll most likely want to use it on that platform. These days when .NET is becoming more platform-independent, it could maybe be used on Linux/Max also although I haven't tried it yet. Most likely I'd just use Java on those platforms.

Java

Java is similar to C# (it's more proper to say it the other way around since C# was created after and was greatly inspired by Java) although it's more bare-boned. You could argue there's less batteries included although that'd probably depend on the developer.

You can run trading programs written in Java on Linux, Windows, and Mac.

C++

C++ is the oldest programming language here, and the most low-level. Its advantages are speed and the increased control.

There several reasons why I don't recommend using C++ for the use with TWS API. The first is that the development speed is much slower. With increased control comes increased needed time investment to do an equivalent trading program in C++ compared to other TWS-supported programming languages on the list.

But what if speed is really important to you and you are prepared to invest more time and resources to be fast? In that case choosing Interactive Brokers might not be the best option. Its servers are probably located somewhere far away from where your server is located, and the time needed for the data to travel over the network is probably much larger than that of running the trading program.

I'd only use C++ in the case where a heavy algorithm needs to be run in real-time, calculating, for example, some large matrixes. Although even in that case it'd probably be better to use single-threaded Java with Garbage Collector disabled, or something like that.

VB

You don't want to use VB. Its variants take 3 out of 4 places for the most dreaded languages in the Stack Overflow's 2017 Developer Survey. Use it only if its the only language you know.

Python

Although still only supported by the TWS API Beta at the time of this writing, all functionality is implemented. Python is relatively easy-to-use programming language in which you can write code fast. Since it's not statically-typed, there are limitations to how much IDE can help you with hints, but the advantage is that less code is required.

It'll take much less time to write an equivalent program in Python than in any other language on the list (supposed you know all the languages equally well). It's quite easy to make trading program and a backtest that use the same trading-logic code, which is what you want to do if you want to be sure the backtest works correctly. This would take more time in C# or Java.

It runs on Windows, Linux, and Mac.

The Winner

In most cases, probably 80%+, I'd recommend Python, especially once it comes out of Beta. Smaller and middle-sized trading programs will benefit the most from the easiness-of-use that comes with Python.

But I would not recommend Python when the project is going to be larger. I'd say that is more than around 10,000 lines of code. With the increased size the lack of static types can hurt the ability to write robust code. It's easier to introduce bugs. After a certain project size, it just takes longer to read/write Python code than when using one of the more strict programming languages. I'd choose C# when using Windows and Java when using other platforms.

In the end, you want to choose the programming language you know best, or one you want to learn. If you know all equally well or know none, I'd recommend what I've written above - probably Python, or C#/Java for large projects.


What is the chance of losing money in the stock market?

Posted on Mon 23 January 2017 in finance • Tagged with investing, savings, money

The first step of achieving good returns in the stock market is not losing money. What is the percentage chance of losing money in the stock market?

That depends on the investment style and can be hard to quantify. If you hold a small number of companies in the portfolio, the chance of losing money is greater because if any single company performs badly, the effect on the portfolio will be large.

But say you invest in a large number of companies, say in the companies of some stock index. We cannot be certain of any performance of the selected companies in the future. What we can do is we can look at how the stock market index performed in the past.

To do this, I wanted to have as large amount of data as possible. The most data I could find, is for a stock index S&P 500. The index consists of 500 large companies from the United States (large based on the market capitalization) and in the analysis, data from 1871 to 2016 is used. How the largest 500 companies' stock price move is a good-enough heuristics of how the stock market moves in the United States in general. The source of all data is multpl.com.

Let's say you invested your money into the S&P 500 index at some random year after year 1871, on the January 1, for X number of years. What's the chance your return is positive or negative?

Let's look at the data. Here we suppose all the dividends are paid on the January 1 and reinvested immediately. No tax is applied.

SPY chance of losing money, no inflation

We can see that chance of being profitable depends a lot on how long you are invested. As the common wisdom says, investing in the stock market should be done for a longer term.

The data above used nominal prices, i.e. was not adjusted for inflation. The chart above should be therefore relevant to you if you consider either investing in the stock market or holding cash.

If you consider either investing in the stock money or keeping money in some sort of a financial vehicle (i.e. TIPS, or perhaps a very diverse commodity index), I've also tested the same thing but discounted returns for each year by whatever the inflation (US CPI) was (dividends included).

The chance of beating that vehicle, investing for X years, is shown in the chart below.

SPY chance of losing money, inflation


A short guide for new VIM users

Posted on Sat 10 December 2016 in vim • Tagged with vim

VIM is a text editor that makes writing code MUCH more enjoyable. After using it for a long time it's almost hard to believe how people can write code without it. It's like driving a new BMW opposed to an old Peugeot. Everything is just smooth and fast. And unlike BMW, it's free! Watching another programmer write code without using VIM is almost like watching someone type using two fingers (one with each hand). You just want to say: give me the keyboard, I'll do it for you. The programmer uses UP, DOWN, LEFT and RIGHT keys, reaches for the mouse a lot, repeats certain actions a lot and is most likely very efficient and clever with the use of HOME, END, PGUP and PGDOWN keys. But compared to VIM it's all slow and awkward.

The best thing about VIM is that you don't even need to use the original program! You just install a VIM plugin for your favorite editor (such plugin exists for all major editors, including Visual Studio, all JetBrains editors, Atom, Sublime, ...) and you get all the functionality of the original VIM editor. What's great here is that using VIM gives you a consistency across all the editors. You don't need to remember as much specific stuff for each editor (i.e. what's the shortcut for Redo in this IDE, again?) - VIM is the same in all of them.

Modes

When writing code (or some other text) in a regular editor the key, the keys you press appear on the screen right away. Using VIM this will happen only if you are in so called insert mode. When in this mode, VIM behaves as any other editor. After VIM has just started, it is in normal mode. If you simply want to write some text, press i to enter insert mode. Now you can type anything you want. To exit insert mode and go back to normal mode, press ESC.

Moving around

So why would is this normal mode even needed if we use keyboard to type characters on the screen and the only way we can do that is by being in the insert mode. Well, VIM makes the keyboard much more versatile tool. For example, when in normal mode, you can use keys H, J, K and L in order to move LEFT, DOWN, UP and RIGHT, respectively. That's very useful since it removes the need to always move your hand to the arrow keys. It takes some time to get used to but once you know it, it's hard to go back. When the hands are in the typing position (left index finger on F and right index finger on J) it's much more convenient to simply press ESC (which I even virtually switched with CAPS LOCK to have it closer to the standard position) and use H, J, K and L.

Moving to the start of the next word is done by pressing W and moving to the start of previous word by pressing B. It's similar to pressing CTRL + LEFT/RIGHT when using regular editor, without the need to move right hand away from natural position and pressing two keys at the same time.

Another thing I use a lot is pressing { and } which will jump the cursor to previous and to next paragraph (in normal mode of course) and pressing 0 and $$ which move the cursor to the beginning or end of the line you are currently on.

I think that this alone is worth using VIM when writing code or anything else.

Copying, deleting and pasting lines

When writing code it's common to copy a line and paste it somewhere else. Copying a line is done by pressing YY (i.e. pressing Y twice). After that you can move anywhere else and with pressing P the line will be pasted under the line your cursor is on.

Deleting a line is done by pressing DD. We can imagine it's similar to cutting a line as you can still paste it using P.

Undo / Redo

Undoing the last command (i.e. inserting, deleting, pasting, etc.) is done by pressing U. The redo command is CTRL + R.

Selecting lines

Another very useful command is SHIFT + V which starts to select lines. After moving up or down (preferably with K or J) you press Y to copy them. Now you can move to some other place and press P to paste copied lines (use D instead of Y to delete them).

Saving positions and jumping

Wherever your cursor is located you can press M followed by any character an the position will be saved. After that you can press ' followed by the character you used previously and your cursor will jump to saved position. Very useful!

Searching

Searching is done by simply pressing /, typing the word to search for and pressing ENTER. The cursor is moved to the first found word. After that, pressing N will move the cursor to the next found word (and SHIFT + N will move it to the previous found word).

Replacing

Often you need to replace all words with some other words in a given text. First, you select the text (by using SHIFT + V, remember?) and then typing this (without the < and >): :s/<text to search>/<text to replace it with>/g. It might sound complex but once you get used to it it's so much faster and easier than using the default Find & Replace functionality of whatever IDE you are using at this moment.

Other small things

Pressing SHIFT + I will open insert mode in the beginning of the row and SHIFT + A will open insert mode at the end of the row.

Repeating commands machine

Repeating commands is where VIM really shows its power. Whatever command you've seen so far, you can save its execution sequence and repeat it. This is huge! First you record a command or a sequence of commands by first pressing Q<some key, say A>, executing the commands, and then pressing Q to end the execution and save it. Then you can replay it by pressing <number of times to repeat>@a.

Let's look at an example. Say I have these lines in my editor and I want to add parenthesis around them, with a comma on the right and a tab on the left.

Marlon Brando Al Pacino James Caan Richard S. Castellano Robert Duvall Sterling Hayden

I could do it by hand or I could utilize the great power of VIM and do it singlehandedly, wizard-style. I'll do it by first moving to the first line and pressing QA - that will start recording my commands under the key A. Then I do SHIFT + I to move to the beginning of the line and open normal mode, followed by pressing TAB and ". Then I press ESC to exit insert mode and return to normal mode and do SHIFT + A to open insert mode at the end of the line. Next, I press " and ,. I exit insert mode again and press J to move one line down. My execution sequence is finished so I complete it by pressing Q.

Now VIM has remember the whole execution and will execute it any number of times. I simply do [email protected] and the execution will be repeated 5 times, fixing all my lines.

How to learn all this?

There's no need to learn everything to start using VIM. I think that VIM has major advantages even when you use only a small subset of the provided functionality. Simply start using it by installing a VIM plugin for your code editor and using mostly insert mode and maybe H, J, K and L (the keys to move in different directions, remember?) in the normal mode instead of the arrow keys. After you get a grasp of that, add some other thing that sounds interesting. It's like driving a car - at first it's hard and complex, after a few months it's completely automatic and you don't even think about it.


Keyboard tricks to write code faster

Posted on Mon 31 October 2016 in typing • Tagged with typing, fast

Using English keyboard layout

Using any keyboard layout other than English may be less optimal for programming, at least that was my experience when compared to using Slovenian and German keyboard layouts to write code. Switching to the English layout made a great difference for me. It's simply faster to reach commonly used characters such as []{};:,@""<>= compared some (maybe all) other layouts. I've trained myself to always use English keyboard when programming and when writing in English language in general (such as now).

When writing in Slovenian language I switch the keyboard layout to Slovenian and continue seamlessly as nothing changed. Then only keys that I sometimes use wrong on Slovenian layout are letters Y and Z which are for some reason switched in Slovenian keyboard layout.

Try using VIM

The next level mastery is not only to stop reaching for a mouse but also stop reaching for the movement keys (UP, DOWN, LEFT and RIGHT). They are also quite far away from the keys you use to do the actual writing. When programming we use movement keys a lot and minimizing or entirely removing the need to access them saves time and thousands of unnecessary hand movements.

Switch Caps Lock and Esc keys

Since I press Esc much more than Caps Lock, I switched the two keys (functionally, not physically). I use software called SharpKeys to do that. It's much more enjoyable to use the keyboard like that.

I also considered switching Backspace with some other key but haven't so far found a good replacement. I thought about Right Shift, which I don't use a lot, but it's not that of a good improvement.


Using PyCharm without a mouse

Posted on Sat 29 October 2016 in pycharm • Tagged with pycharm, shortcuts

I've been always fan of minimizing mouse/touchpad usage when programming. There is a certain kind of satisfaction when you are able to do everything with the keyboard. You get the feeling of being agile, swift. It's a sign of mastery. There's no unnecessary movement and unnecessary time and muscle power spent. Watching another programmer use their keyboard solely and be fast doing it makes you respect them more. Watching them grab the mouse a lot and wait for them to find the mouse cursor is sometimes even a little painful.

Run the program

So how do you go about using keyboard more and using mouse less? Well, start off by learning how to run your program with only the keyboard.

How to do that? To run currently open file press CTRL + SHIFT + F10. To run lastly open file, even if you are currently in another file, press SHIFT + F10. To run an arbitrarily chosen file press SHIFT + ALT + F10 to open a dialog box where you select which file to run.

Hide output window

After you run the program output window will appear if it wasn't shown before. Unlike in Visual Studio, this output window will not hide automatically. This can be annoying if you want a large screen space for code. Well, there's a simple shortcut for hiding a window you are in (besides grabbing the mouse and clicking hide window button on top right) and it's SHIFT + ESC. Press that after running the program and output window will close leaving your focus in the code and you can start typing again.

Go to a file

Press CTRL + SHIFT + N and start writing name of a file to jump to it. I've remapped the shortcut to CTRL + T to be equal to the same shortcut in ReSharper (Visual Studio).

Refactoring

The best advantage of PyCharm is its refactoring capability. Being positioned on a variable/function/class and doing CTRL + SHIFT + ALT + T. It gives you lots of refactoring options such as:

  • rename,
  • inline,
  • extract variable, method,
  • change method signature.

These PyCharm refactoring functionalities are not only huge time savers but also, unlike we humans, they don't make mistakes.

Since this shortcut involves four keys, I've remapped it to CTRL + SHIFT + R (same as refactoring using ReSharper in Visual Studio).

Confirm refactoring

When refactoring, PyCharm sometimes requires a confirmation of the refactoring (e.g. after renaming a function). The button you have to click says "Do Refactor" and it's located in another window. This used to annoy me a lot since I didn't know how to press this button via keyboard. But I found the solution. After doing the renaming, simply press ALT + D and the "Do Refactor" button will be clicked. After that press SHIFT + ESC to close the refactoring window.

Toggle full screen

You can add your own shortcut to toggle full screen if you go to Settings -> Keymap (shorcut to doing that is pressing CTRL + ALT + S and typing Keymap) since there is no default shortcut.

But you can still do it with your keyboard pretty fast by pressing ALT + V to open the View menu and pressing up arrow and Enter.

I got used to that even though it's not the completely fastest way although it's pretty fast.

You can't do everything with a keyboard

I don't use solely keyboard for every single thing in PyCharm. I still use the mouse to scroll up and down in the output window. I use the mouse to open a file for which I am not exactly sure what its name is but I know where it's located in the folder hierarchy. My goal is not to never use the mouse but to not use it when there's a fast and comfortable alternative. There is a lot of these kinds of shortcuts and they keep surprising you if you look for them.


How to fix mouse offset bug in VMWare Workstation under Windows 8.1 host

Posted on Fri 12 February 2016 in vmware • Tagged with vmware, bug

There’s a bug in mouse behavior that happened to me in VMWare Workstation under Windows 8.1 host with Ubuntu 14.04 as client. When hovering mouse over something and/or selecting something by dragging, the mouse moved it’s position by some offset. See the GIF below.

VMWare bug

How to fix it?

  • Locate VMWare Workstation, right click and open Properties.
  • Under Compatibility check Disable display scaling on high DPI settings.
  • Restart VMWare Workstation.

The mouse should now behave normally.


[C#] Why it’s better to have short-lived DataContext

Posted on Wed 20 August 2014 in C# • Tagged with c#, datacontext

When you work with SQL databases in your C# application, one way is to use LINQ to SQL framework. The way to use LINQ to SQL is to create an instance of DataContext for your specific database and use it to retrieve, insert, update and delete data.

If the project is not too large, you could have all your dealings with database abstracted in one class. That puts all your database logic in one place. You also can easily mock this class (as opposed to mocking DataContext). Let’s call this DatabaseOperations.

When having a class which contains methods that do work the database, two ways of dealing with DataContext come to mind.

public class DatabaseOperations : IDatabaseOperations
{
    private readonly MyDatabaseDataContext _context;

    public DatabaseOperations(MyDatabaseDataContext context)
    {
        _context = context;
    }

    public List<User> GetUsers()
    {
        // Retrieve users by using _context.
    }

    public List<Role> GetRoles()
    {
        // Retrieve roles by using _context.
    }

    public List<Group> GetGroups()
    {
        // Retrieve groups by using _context.
    }
}

Another way would be without the context instance as a field. Here each method creates and closes its own context. Sample code:

public class DatabaseOperations : IDatabaseOperations
{
    private readonly string _connectionString = "myConnStr";

    public List<User> GetUsers()
    {
        using (var context = new MyDatabaseDataContext(_connectionString))
        {
            // Retrieve users by using context.
        }
    }

    public List<Role> GetRoles()
    {
        using (var context = new MyDatabaseDataContext(_connectionString))
        {
            // Retrieve roles by using context.
        }
    }

    public List<Group> GetGroups()
    {
        using (var context = new MyDatabaseDataContext(_connectionString))
        {
            // Retrieve groups by using context.
        }
    }
}

At first, the second approach may seem to be more wasteful as each function has to create the context. That creates some amount of duplicated code. However, there are two reasons why this second approach is preferred.

The first reason is that in the first example the instance of a class DatabaseOperations can live a long time. You probably do not create new instance of the class each time you call function inside. Long lived DatabaseOperations instance means also that the injected DataContext lives long time. The DataContext, however, is supposed to be used on only small units of work. The reason is that in the first example, one instance of DataContext has to track the changes that happen in returned objects.

More info in the official documentation:

In general, a DataContext instance is designed to last for one “unit of work” however your application defines that term. A DataContext is lightweight and is not expensive to create. A typical LINQ to SQL application creates DataContext instances at method scope or as a member of short-lived classes that represent a logical set of related database operations.

The second reason, which is important from the design point of view, is that in first example, DataContext has to be passed in via constructor. However, the DatabaseOperations class behaves like a Service Object and in order to use dependency injection, Service Objects should take as constructor parameters only other Service Objects. DataContext is more of a Value Object which does not belong in the parameter list. The tree of Service Objects cannot be created at the start of the program.

For more info see this blog article by Miško Hevery (who is the author of JavaScript framework AngularJS).

You could, of course, also have DataContext as a parameter in every function. That would allow your class to be a Service Object which has no Value Objects in the constructor parameter list. However, the first reason would still be violated if that DataContext is long-lived or, as the documentation says, used for more than one “unit of work”.

In order to somehow mitigate the duplication of code by creating a new DataContext in every method, you might be able to use the trick from this StackOverflow answer.

That might or might not be useful for you, depending on where you store and how you obtain the connection string. For example, if you have it stored as a static class field, then it is appropriate. If you have to get the connection string from a file or another database, then it might get a bit complicated (in my last project, I decided against it and just used the simple approach shown in the second example).