Thursday, July 10, 2014

[Solved] Scm-Manager high cpu load

We use git. Our repo size is 2 GB with 60 K files and 14 K commits.

Suddenly, scm-manager (1.38) was using all the CPU.

And we had strange failures in TeamCity checkouts like :
remote: internal server error
fatal: protocol error: bad pack header
The problem was that our central bare repository had never been garbage collected (with git gc). Even running git gc was failing with the error
warning: packfile ... cannot be accessed
fatal: failed to read object ...: Too many open files
error: failed to run repack
Here is what I did :
  • bare clone the repo on another machine
  • run git gc
  • bare clone the previous repo to the original machine
  • run git gc
  • switch last repo with the broken one (et voilà)
HTH

Ref : https://groups.google.com/forum/#!topic/git-users/6XChKIqdG_U

Sunday, June 15, 2014

Test Driven FizzBuzz

Here is my solution when I apply TDD on FizzBuzz using C# :
class FizzBuzz
{
    private readonly List<Tuple<int, string>> rules;
    public FizzBuzz()
        : this(new Tuple<int, string>[0])
    {
    }
    public FizzBuzz(IEnumerable<Tuple<int, string>> rules)
    {
        this.rules = rules.ToList();
    }
    public string Convert(int i)
    {
        var label = string.Join("", rules.Where(r => i % r.Item1 == 0).Select(r => r.Item2));
        return label.Length > 0 ? label : i.ToString();
    }
}

[TestFixture]
public class FizzBuzzTest
{
    [TestCase(1, "1")]
    [TestCase(2, "2")]
    [TestCase(3, "3")]
    public void TestNoRule(int i, string value)
    {
        Assert.AreEqual(value, new FizzBuzz().Convert(i));
    }
    [TestCase(1, "1")]
    [TestCase(2, "2")]
    [TestCase(3, "Fizz")]
    public void TestFizz(int i, string value)
    {
        Assert.AreEqual(value, new FizzBuzz(new[] { new Tuple<int, string>(3, "Fizz") }).Convert(i));
    }
    [TestCase(1, "1")]
    [TestCase(2, "2")]
    [TestCase(3, "3")]
    [TestCase(4, "4")]
    [TestCase(5, "Buzz")]
    public void TestBuzz(int i, string value)
    {
        Assert.AreEqual(value, new FizzBuzz(new[] { new Tuple<int, string>(5, "Buzz") }).Convert(i));
    }
    [TestCase(15, "FizzBuzz")]
    public void TestFizzBuzz(int i, string value)
    {
        var rules = new[] {
            new Tuple<int, string>(3, "Fizz"),
            new Tuple<int, string>(5, "Buzz"),
        };
        Assert.AreEqual(value, new FizzBuzz(rules).Convert(i));
    }
    [Test] public void TestFizzBuzz1_100()
    {
        var rules = new[] {
            new Tuple<int, string>(3, "Fizz"),
            new Tuple<int, string>(5, "Buzz"),
        };
        var fizzBuzz = new FizzBuzz(rules);
        for (int i = 1; i < 100; i++)
            Console.WriteLine(fizzBuzz.Convert(i));
    }
}
See also :

Saturday, April 12, 2014

Samsung vs Google Calendar

On my Samsung Galaxy S3, I have two calendar applications : Google Calendar and S Planner (S Calendrier in French). Both applications are able to display all phone calendars :
  • My Calendar : To display contact birthdays
  • Samsung calendar
  • Google calendar(s) : your own Google calendar(s) and your contacts'
If you sync these calendars, this will only keep in sync your phone calendars with the various clouds (Google, Samsung, Facebook, etc.). But this will not sync calendars between one another.
Sometimes, Samsung Calendar becomes the default when you create new events. I have seen lots of people on the web looking for ways to import/export from Samsung Calendar to Google, or to sync them...
Once an event is created, AFAIK, it is not possible on my phone to move the event to another calendar.
I have found a solution : with Kies Air opened in your computer browse, you can move events to another calendar.
How to proceed :
  • (phone) Open Samsung Kies Air app
  • (computer) Open Kies Air in your browser : http://phone.ip:8080
  • (computer) Enter PIN displayed on your phone
  • (computer) Open Calendar and edit events to move them to other calendars
HTH

Saturday, August 17, 2013

GitExtensions ContextMenuHandlers

Here is a thread about duplicated menu entries in GitExtensions Explorer handlers.

Saturday, August 3, 2013

Move User Profile Folder

I had to move my user profile folder to another drive (because I could'nt wait for gparted to move my data). I am using Windows 7 Professionnal (in French). Context :
  • Moved profile : ded
  • Current (old) profile path : c:\Users\ded
  • Future (new) profile path : D:\ded
Here is what I did :
  • Create an(other) admin account (for example: root)
  • Close current ded session
  • Create d:\ded
  • Fix d:\ded ACL (you would need to uncheck include parent security)
My profile folder security is :
  • Administrators : Full Control, recursively
  • System : Full Control, recursively
  • ded : Full Control, recursively
  • root : Full Control, recursively
  • HomeUsers : see below
(It seems that root what added automatically under the hood)

HomeUsers permissions are :
  • Traverse folder / execute file
  • List folder / read data
  • Read attributes
  • Read extended attributes
  • Read permissions
HomeUsers permissions only apply to this folder only (not sub-folders nor files). French translation :
  • Parcours du dossier/exécuter le fichier
  • Liste du dossier/lecture des données
  • Attributs de lecture
  • Lecture des attributs étendus
  • Autorisations de lecture
Let's continue :
  • Copy all files with robocopy (keep permissions, etc.)
robocopy c:\users\ded d:\ded /e /copyall /sl /xj /np /nfl /r:1
  • Restart robocopy to find what failed
robocopy c:\Users\ded d:\ded /e /copyall /sl /xj /np /nfl /ndl /r:1 /w:1 /x
In my case I only had problems with :
  • junctions (not handled by robocopy)
  • Some tmp files (ignored)
  • Cardspace files (access denied!)
  • Google drive folder (denied)
To find junctions run :
dir c:\users\%username% /al /s
I used the following script to create junctions in my new profile folder. *BEWARE !* This script is for a French O/S.
@echo off
:: é = ‚
:: è = Š
setlocal
set new_home=d:\%username%
call :mk_junction "Application Data" "AppData\Roaming"
call :mk_junction "Cookies" "AppData\Roaming\Microsoft\Windows\Cookies"
call :mk_junction "Local Settings" "AppData\Local"
call :mk_junction "Menu D‚marrer" "AppData\Roaming\Microsoft\Windows\Start Menu"
call :mk_junction "Mes documents" "Documents"
call :mk_junction "ModŠles" "AppData\Roaming\Microsoft\Windows\Templates"
call :mk_junction "Recent" "AppData\Roaming\Microsoft\Windows\Recent"
call :mk_junction "SendTo" "AppData\Roaming\Microsoft\Windows\SendTo"
call :mk_junction "Voisinage d'impression" "AppData\Roaming\Microsoft\Windows\Printer Shortcuts"
call :mk_junction "Voisinage r‚seau" "AppData\Roaming\Microsoft\Windows\Network Shortcuts"
call :mk_junction "AppData\Local\Application Data" "AppData\Local"
call :mk_junction "AppData\Local\Historique" "AppData\Local\Microsoft\Windows\History"
call :mk_junction "AppData\Local\Temporary Internet Files" "AppData\Local\Microsoft\Windows\Temporary Internet Files"
call :mk_junction "AppData\Roaming\Microsoft\Windows\Start Menu\Programmes" "AppData\Roaming\Microsoft\Windows\Start Menu\Programs"
call :mk_junction "Documents\Ma musique" "Music"
call :mk_junction "Documents\Mes images" "Pictures"
call :mk_junction "Documents\Mes vid‚os" "Videos"
endlocal
goto :eof

:mk_junction
set link=%1
set target=%2
set link="%new_home%\%link:~1,-1%"
set target="%new_home%\%target:~1,-1%"
echo %link% -^> %target%
if exist %link% (
  echo found %link%, skipped
  goto :eof
)
mklink /J %link% %target%
::icacls %link% /deny Everyone:(S,RD) /L
icacls %link% /deny "Tout le monde":(S,RD) /L
icacls %link% /setowner SYSTEM /L
attrib +H +S +I %link% /L
goto :eof
For cardspace files, I suspected a mismatch between Administrateurs (French, unknown group) and Administrators (English, valid group) accounts. I managed to move or copy the files with cygwin mv or cp. Afterwards, I just did *attrib +h* on cardspace folder and files (CardSpaceSP2.db and CardSpaceSP2.db.shadow).

I ignored Google Drive, Google recreate it automatically (it wouldn't use a copy of the original folder).

Final step :
move c:\users\ded c:\users\ded.old
mklink /j c:\users\ded d:\ded
And fix d:\ded permissions (same as above).

Now I reopen a session with ded users with my new user profile folder.

HTH

Friday, May 10, 2013

Saving OLEObject Content To File

It is possible with OLE to embed files in Excel workbooks and saves them back to disk (I know OLE is not cutting edge technology). To embed some file :
  • Insert ribbon menu
  • Object (in Text)
  • From file tab
  • (Browse to file)
If your file is another Office document, saving it back to disk is trivial. But here I want to embed any file (e.g. txt, xml). Anton post gave me the directions. I pushed further the analysis and eventually reversed engineered the OLEObject MemoryStream content :
0x2 0x0header
string\0file name
string\0file path
0x0 0x0 0x3 0x0(native header ?)
inttemp file path length
string\0temp file path
intcontent length
bytescontent
inttemp file path utf16 length
bytestemp file path utf16
intfile name utf16 length
bytesfile name utf16
intfile path utf16 length
bytesfile path utf16
Note : header is different for non Package OLE objects like Office documents or pdf.

In this sample program, I load some excel workbook and for each embedded ole object, I display its name and its content :
[STAThread]
static void Main(string[] args)
{
    var excel = new Application();
    try
    {
        Workbook workbook = excel.Workbooks.Open(@"D:\Classeur1.xlsx");
        foreach (Worksheet worksheet in workbook.Worksheets)
        {
            foreach (OLEObject ole in worksheet.OLEObjects())
            {
                Console.WriteLine("name : {0}", ole.Name);
                if (ole.progID == "Package")
                {
                    string content = ole.GetContent();
                    if (content != null)
                        Console.WriteLine(content);
                }
            }
        }
        workbook.Close();
    }
    finally
    {
        excel.Quit();
    }
    Console.Write("Press a key...");
    Console.Read();
}
The progID has "Package" value when the embedded content is not standard OLE content. The GetContent extension method gets a MemoryStream from the OLEObject and loads the content from the stream :
static class OLEExtensions
{
    public static string GetContent(this OLEObject ole)
    {
        ole.Copy();
        System.Windows.Forms.IDataObject data = System.Windows.Forms.Clipboard.GetDataObject();
        object obj = data.GetData("Native");
        System.Windows.Forms.Clipboard.SetDataObject("");
        var ms = obj as MemoryStream;
        if (ms != null)
            return ms.GetOLEContent();
        return null;
    }
}
We copy the OLE object to the clipboard to get the MemoryStream. The STAThread attribute is required in Main method to avoid some NullReferenceException when calling GetData method. The GetOLEContent extension method extracts the content from the stream thanks to the reverse engineered stream structure :
static class OLEStreamExtensions
{
    public static int ReadHeader(this MemoryStream ms)
    {
        var header = new byte[2];
        int read = ms.Read(header, 0, header.Length);
        if (read != header.Length)
            throw new FormatException("End of stream while reading header");
        if (header[0] != 2 || header[1] != 0)
            throw new FormatException("Bad header");
        return read;
    }
    public static string ReadString(this MemoryStream ms)
    {
        var sb = new StringBuilder();
        while (true)
        {
            int b = ms.ReadByte();
            if (b == -1)
                throw new FormatException("End of stream while reading string");
            if (b == 0)
                return sb.ToString();
            sb.Append((char)b);
        }
    }
    public static int ReadInt(this MemoryStream ms)
    {
        var bytes = new byte[4];
        int read = ms.Read(bytes, 0, bytes.Length);
        if (read != bytes.Length)
            throw new FormatException("End of stream while reading int");
        return BitConverter.ToInt32(bytes, 0);
    }
    public static byte[] ReadBytes(this MemoryStream ms, int count)
    {
        var bytes = new byte[count];
        int read = ms.Read(bytes, 0, count);
        if (read != count)
            throw new FormatException("End of stream while reading bytes");
        return bytes;
    }
    public static string GetOLEContent(this MemoryStream ms)
    {
        ms.ReadHeader();
        string name = ms.ReadString();
        string path = ms.ReadString();
        int reserved = ms.ReadInt();
        if (reserved != 0x30000)
            throw new FormatException(string.Format("Unexpected reserved bytes : got {0} but expected {1}", reserved.ToString("x"), 0x30000.ToString("x")));
        int tempLength = ms.ReadInt();
        string tempPath = ms.ReadString();
        if (tempPath.Length + 1 != tempLength)
            throw new FormatException(string.Format("Mismatch between temp length {0} and temp full path length {1}", tempLength, tempPath.Length + 1));
        int contentLength = ms.ReadInt();
        byte[] content = ms.ReadBytes(contentLength);
        int delta = sizeof(int) * 3 + (name.Length + path.Length + tempPath.Length) * 2;
        if (ms.Length != ms.Position + delta)
            throw new FormatException("Unexpected end of file");
        return UTF8Encoding.UTF8.GetString(content);
    }
}
This code uses Excel but it might work with any Office document (Word, Powerpoint...). Of course, you should have installed the PIA. I have validated this code with xml and txt files. I use Excel 2007 SP3 MSO.

Edit : You can refactor this code to extract the name and the bytes of each embedded object to be able to save the contents to files :
class OLEContent
{
    #region Fields
    private readonly string name;
    private readonly byte[] content;
    #endregion
    public OLEContent(string name, byte[] content)
    {
        this.name = name;
        this.content = content;
    }
    public string Name { get { return name; } }
    public byte[] Content { get { return content; } }
}
...
public static OLEContent GetOLEContent(this MemoryStream ms)
{
    ...
    return new OLEContent(name, content);
}
...
OLEContent content = ole.GetContent();
if (content != null)
    File.WriteAllBytes(Path.Combine(tempPath, content.Name), content.Content);
...
This new version can also save images (like jpg) to disk.

Sunday, March 31, 2013

From Google Reader To Tiny Tiny RSS

In early 2000s, I migrated from FeedReader application to Google Reader. Now it's time to move on, again.
I have chosen Tiny Tiny RSS because :
  • It was in Slashdot comments
  • It has both a Web interface and an Android app
  • It is a very active project
  • I can host it myself (it's not in some cloud)
  • Articles will not be marked read after 30 days
  • I can disable article purge
Hosting
Tiny Tiny RSS works fine with WampServer but I do not allow connections from internet to my local home private network.
I could not use the following web hosting providers :
  • free.fr : no support for PHP 5.3
  • olympe : doc is dead, do not know how to upload files
  • 000webhost : ftp is deadly slow, tt-rss does not work (uname, not writable errors)
  • byethost : dojo not defined error, could not load feeds
  • Toile Libre : dead
I have chosen hostinger :
  • ttrss works ! (PHP 5.3.20 and you can enable PHP 5.4)
  • small .p.ht url suffix
PHP has open_basedir restrictions but :
  • you can use this patch
  • You must use the 'resolved' url once all redirections have been resolved
Sample feeds resolved by firefox :
  • The Daily WTF : http://thedailywtf.com/rss.aspx gives http://syndication.thedailywtf.com/TheDailyWtf
  • PC Inpact : http://www.pcinpact.com/rss/news.xml gives http://pcinpact.com.feedsportal.com/c/35178/f/652880/index.rss
Edit 2013.04.01 : hostinger lets you add cron jobs to update feeds.
Edit 2013.04.11 : 10 days later, I got an official answer from hostinger support saying that cron jobs only support PHP 5.2. Too bad...