The Sugar House was erected in 1763 on Prince Street (as Rose Street was formerly called) by Henry Cuyler. A wealthy Dutch family, the Cuylers were able to construct an impressive stone factory building. Undoubtedly, with six stories plus a basement, it was one of the largest buildings in the British Colonies. Original photographs taken before the demolition of the structure in 1892 confirm the medal's rendering: in the upper story of the house, the letters "BRC"—representing the initials of Henry Cuyler's son, Barent Rynders Cuyler—were worked into the façade in contrasting brick.

It is perhaps not surprising to find that Edward Groh, inspired by the history and myths of the Rhinelander Sugar House, commissioned this medal in 1892, the year of the demolition. The ANS records show that he had intended the subject to be part of Augustus Sage's Series of Historical Tokens relating to the Revolution.

In the year 1859. Mer Aug. B. Sage issued his Series of Historical Tokens and I suggested to him to include the Rose St Sugar house in the series which he agreed to do if I would furnish him with the drawing and before I could do so the series was discontinued. To my regret as I wished to see the building perpetuated on a medal, thirty three years later and before its demolition, I concluded to gratify my wish and Mr E. Sigel executed the work, The word Founded was not intended as I ordered it to be "erected" and the error is due to Mr Sigel to whom I exhibited the Sage Medalet of Liberty St Sugar house and he got the words mixed up. Edward Groh