[Open.ogc] Deleted layers still exist?
Micah Wengren
micah.wengren at noaa.gov
Wed Sep 25 18:38:00 UTC 2013
Hi John,
I was just about to email you back on this, because I got an update from
support that unfortunately that feature isn't implemented in GeoServer
after all. Apparently he was mistaken because it is implemented in
GeoTools (the java library that GeoServer uses behind the scenes), but
not leveraged in the GeoServer code.
I asked what it might take to get that added, so stay tuned. I'm
expecting an answer of either 1) you can do it yourself if you want (a
benefit of using an open source product), 2) it might get added at some
point, file a feature enhancement request:
http://jira.codehaus.org/browse/GEOS, or 3) if you have some money, you
can fund it to be developed by us.
We did #3 when this project got off the ground originally, we funded
them to develop the workspace access restrictions that allows multiple
users of the same system, but money has been harder to come by lately
(if NMFS has some extra lying around, let me know!).
In the meantime, if it's essential to delete the old schemas for those
tables, I can put a request in to the WOC to see if they can do it
manually in the database during a maintenance period. Let me know.
Micah
On 9/25/2013 2:15 PM, John Kennedy - NOAA Affiliate wrote:
> Okay, I deleted 'communities_pgs' using the Web UI and then used
> hpcc_data_requests.py script to load the data. Same result as before.
> The old schema remains and none of the new schema is added.
>
>
> John F. Kennedy, PhD, CPG
> GIS Specialist
> Contractor: Caelum Corp.
> NOAA Fisheries Office of Science and Technology
> 1315 East-West Highway
> SSMC3 Rm 9518
> Silver Spring, MD 20910
> 301-427-8149
>
> Comments? Have I been of assistance? Please let my supervisors know. :~)
> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov
> <mailto:Tim.Haverland at noaa.gov>>
> Hussain Jabalpurwala - NOAA Affiliate <hussain.jabalpurwala at noaa.gov
> <mailto:hussain.jabalpurwala at noaa.gov>>
>
>
> On Wed, Sep 25, 2013 at 11:39 AM, Micah Wengren
> <micah.wengren at noaa.gov <mailto:micah.wengren at noaa.gov>> wrote:
>
> Hi John,
>
> I am working on troubleshooting this issue with OpenGeo/Boundless
> support. I've gotten the response back that GeoServer should be
> able to modify an underlying table schema when a new shapefile is
> uploaded with changes in the attribution, but I think based on
> your evidence more testing of this is needed.
>
> I asked whether it made a difference if the layer (and associated
> featuretype) were currently published when the new data upload
> happened, or if they should first be deleted. If you get a
> chance, could you test with the same layer you used below
> 'communities_pgs' by first deleting the layer via Web UI and then
> doing the data upload (incl new schema) with the
> hpcc_data_requests.py script?
>
> Micah
>
>
> On 9/24/2013 4:14 PM, John Kennedy - NOAA Affiliate wrote:
>> Interesting question. I did a preview and the features do appear
>> to be overwritten in the layer. But the "new" attributes are not
>> loading as the column names defer between the old and the new
>> shape file. I checked a number of polygons and got blank values
>> as shown below.
>>
>> Inline image 1
>>
>> John F. Kennedy, PhD, CPG
>> GIS Specialist
>> Contractor: Caelum Corp.
>> NOAA Fisheries Office of Science and Technology
>> 1315 East-West Highway
>> SSMC3 Rm 9518
>> Silver Spring, MD 20910
>> 301-427-8149 <tel:301-427-8149>
>>
>> Comments? Have I been of assistance? Please let my supervisors
>> know. :~)
>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov
>> <mailto:Tim.Haverland at noaa.gov>>
>> Hussain Jabalpurwala - NOAA Affiliate
>> <hussain.jabalpurwala at noaa.gov
>> <mailto:hussain.jabalpurwala at noaa.gov>>
>>
>>
>> On Tue, Sep 24, 2013 at 4:05 PM, Micah Wengren
>> <micah.wengren at noaa.gov <mailto:micah.wengren at noaa.gov>> wrote:
>>
>> Sure, no problem. So if you take your new shapefile with
>> modified columns and try to upload it to the original layer
>> name 'communities_pgs', can you tell for sure that it does
>> not overwrite data in the columns that haven't changed from
>> the old version to the new? It should be doing that at
>> least, and discarding any new columns, as well as probably
>> leaving any original columns that were removed in the new
>> file empty. At least that would be better than not
>> overwriting anything at all.
>>
>> Micah
>>
>>
>>
>> On 9/24/2013 3:47 PM, John Kennedy - NOAA Affiliate wrote:
>>> If the name of the shape file is the same as one of the
>>> layers, then geoserver presents the old data. I took the
>>> shape file I'm attempting to load as communities_pgs and
>>> named it communities_pgs20130924 and it loads as expected
>>> with the new columns and data.
>>>
>>> Thanks for all your help on this issue.
>>>
>>> John F. Kennedy, PhD, CPG
>>> GIS Specialist
>>> Contractor: Caelum Corp.
>>> NOAA Fisheries Office of Science and Technology
>>> 1315 East-West Highway
>>> SSMC3 Rm 9518
>>> Silver Spring, MD 20910
>>> 301-427-8149 <tel:301-427-8149>
>>>
>>> Comments? Have I been of assistance? Please let my
>>> supervisors know. :~)
>>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov
>>> <mailto:Tim.Haverland at noaa.gov>>
>>> Hussain Jabalpurwala - NOAA Affiliate
>>> <hussain.jabalpurwala at noaa.gov
>>> <mailto:hussain.jabalpurwala at noaa.gov>>
>>>
>>>
>>> On Tue, Sep 24, 2013 at 3:32 PM, Micah Wengren
>>> <micah.wengren at noaa.gov <mailto:micah.wengren at noaa.gov>> wrote:
>>>
>>> John,
>>>
>>> You may have hit on an issue with the way our system
>>> works. Can you tell if the attribute values that are
>>> being saved in the database are coming from the new
>>> shapefile you just tried to upload? Could you insert a
>>> test value and try the upload again? I just want to
>>> check that the new data is actually being written when
>>> you specify update=overwrite. It should be, but it
>>> would be good to confirm.
>>>
>>> I don't actually know how GeoServer handles the
>>> overwrite option behind the scenes. I thought it might
>>> drop the underlying database table and recreate, but it
>>> may not be. That would be why the schema is being
>>> preserved. Best way to find out is for me to open a
>>> ticket with OpenGeo. I'll do that later today.
>>>
>>> Micah
>>>
>>>
>>> On 9/24/2013 2:59 PM, John Kennedy - NOAA Affiliate wrote:
>>>> Okay, I now feel a little silly. I did not replace the
>>>> dummy password in my notes file with my LDAP password.
>>>> Shape files and zip files have the same names as the
>>>> layer. I attempted the data load again, using --update
>>>> overwrite, and I got a 201 response. However, when I
>>>> look at the layer in geoserver I see the old data.
>>>> There is a column (property) called GEOID_jlf and that
>>>> column is in the old data layer and not in the new
>>>> layer I'm attempting to load.
>>>>
>>>> Inline image 1
>>>>
>>>>
>>>> John F. Kennedy, PhD, CPG
>>>> GIS Specialist
>>>> Contractor: Caelum Corp.
>>>> NOAA Fisheries Office of Science and Technology
>>>> 1315 East-West Highway
>>>> SSMC3 Rm 9518
>>>> Silver Spring, MD 20910
>>>> 301-427-8149 <tel:301-427-8149>
>>>>
>>>> Comments? Have I been of assistance? Please let my
>>>> supervisors know. :~)
>>>> Tim Haverland - NOAA Federal <Tim.Haverland at noaa.gov
>>>> <mailto:Tim.Haverland at noaa.gov>>
>>>> Hussain Jabalpurwala - NOAA Affiliate
>>>> <hussain.jabalpurwala at noaa.gov
>>>> <mailto:hussain.jabalpurwala at noaa.gov>>
>>>>
>>>>
>>>> On Tue, Sep 24, 2013 at 2:35 PM, Micah Wengren
>>>> <micah.wengren at noaa.gov
>>>> <mailto:micah.wengren at noaa.gov>> wrote:
>>>>
>>>> Ok. Just curious, what are the shapefile names
>>>> you're trying to upload? Are they the
>>>> communities_pts and communities_pgs layers?
>>>>
>>>> Looks like those are gone from the FeatureTypes
>>>> list now
>>>> (https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis.html).
>>>> But they still show up in the list when you go to
>>>> the 'New Layer' page. Not sure I've seen that issue
>>>> before, so might have to file a ticket.
>>>>
>>>> As for credentials, it should only be your NOAA
>>>> LDAP username and password. Those don't get passed
>>>> on to the database, there's a separate way database
>>>> authorization is handled. If you can log in to the
>>>> Web UI, there shouldn't be an issue with the REST
>>>> API, or at least you shouldn't be getting that 401
>>>> response.
>>>>
>>>> Micah
>>>>
>>>>
>>>> On 9/24/2013 2:21 PM, John Kennedy - NOAA Affiliate
>>>> wrote:
>>>>> Thanks. Tomorrow works better for me. As for the
>>>>> credentials, nothing as changed on this end. I was
>>>>> able to load data with the hpcc_data.py script up
>>>>> to yesterday, but I did get an error message when
>>>>> attempting to overwrite the layer/feature type.
>>>>> Now, there maybe credential issues in the database
>>>>> (PostGIS?) but I've no way of knowing.
>>>>>
>>>>> John F. Kennedy, PhD, CPG
>>>>> GIS Specialist
>>>>> Contractor: Caelum Corp.
>>>>> NOAA Fisheries Office of Science and Technology
>>>>> 1315 East-West Highway
>>>>> SSMC3 Rm 9518
>>>>> Silver Spring, MD 20910
>>>>> 301-427-8149 <tel:301-427-8149>
>>>>>
>>>>> Comments? Have I been of assistance? Please let my
>>>>> supervisors know. :~)
>>>>> Tim Haverland - NOAA Federal
>>>>> <Tim.Haverland at noaa.gov
>>>>> <mailto:Tim.Haverland at noaa.gov>>
>>>>> Hussain Jabalpurwala - NOAA Affiliate
>>>>> <hussain.jabalpurwala at noaa.gov
>>>>> <mailto:hussain.jabalpurwala at noaa.gov>>
>>>>>
>>>>>
>>>>> On Tue, Sep 24, 2013 at 2:15 PM, Micah Wengren
>>>>> <micah.wengren at noaa.gov
>>>>> <mailto:micah.wengren at noaa.gov>> wrote:
>>>>>
>>>>> Hi John,
>>>>>
>>>>> That is odd for the 401 response on the data
>>>>> upload. Are you sure the credentials are
>>>>> right? The other two seem ok, depending on if
>>>>> the featuretype existed in the first place or
>>>>> not.
>>>>>
>>>>> I might be able to come by later, but right
>>>>> now I have a deadline to finish something this
>>>>> afternoon. Let me know if you aren't able to
>>>>> get it to work.
>>>>>
>>>>> Micah
>>>>>
>>>>>
>>>>> On 9/24/2013 12:32 PM, John Kennedy - NOAA
>>>>> Affiliate wrote:
>>>>>> Hi Micah,
>>>>>>
>>>>>> Well, I think I'm getting close. I installed
>>>>>> pip, python setup, and request. I used
>>>>>> hpcc_date_requests.py to delete the layer and
>>>>>> the feature type and I get these to responses:
>>>>>> Response: 200,
>>>>>> url=https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis/featuretypes/communities_pgs
>>>>>>
>>>>>> Response: 404,
>>>>>> url=https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis/featuretypes/communities_pgs
>>>>>>
>>>>>> The hpcc_date_requests appears to be
>>>>>> attempting to load data, but the response
>>>>>> back is this:
>>>>>> Response: 401,
>>>>>> url=https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis/file.shp?update=overwrite&configure=all
>>>>>>
>>>>>> Any chance you can come to my cube and have a
>>>>>> look?
>>>>>>
>>>>>> Cheers,
>>>>>> John
>>>>>>
>>>>>>
>>>>>>
>>>>>> John F. Kennedy, PhD, CPG
>>>>>> GIS Specialist
>>>>>> Contractor: Caelum Corp.
>>>>>> NOAA Fisheries Office of Science and Technology
>>>>>> 1315 East-West Highway
>>>>>> SSMC3 Rm 9518
>>>>>> Silver Spring, MD 20910
>>>>>> 301-427-8149 <tel:301-427-8149>
>>>>>>
>>>>>> Comments? Have I been of assistance? Please
>>>>>> let my supervisors know. :~)
>>>>>> Tim Haverland - NOAA Federal
>>>>>> <Tim.Haverland at noaa.gov
>>>>>> <mailto:Tim.Haverland at noaa.gov>>
>>>>>> Hussain Jabalpurwala - NOAA Affiliate
>>>>>> <hussain.jabalpurwala at noaa.gov
>>>>>> <mailto:hussain.jabalpurwala at noaa.gov>>
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 24, 2013 at 10:18 AM, Micah
>>>>>> Wengren <micah.wengren at noaa.gov
>>>>>> <mailto:micah.wengren at noaa.gov>> wrote:
>>>>>>
>>>>>> Hi John,
>>>>>>
>>>>>> How are you doing the attempted data
>>>>>> upload? With the python script? Can you
>>>>>> tell me which python script you are using
>>>>>> if so? If you can, try using the
>>>>>> hpcc_data_requests.py script instead of
>>>>>> the other one (might involve some admin
>>>>>> rights to install python modules on your
>>>>>> workstation). I've update that one more
>>>>>> recently. I'd like to drop the other one
>>>>>> at some point for easier maintenance.
>>>>>>
>>>>>> I think you should be able to upload a
>>>>>> new data set to replace the existing one,
>>>>>> even if it already exists in the database
>>>>>> and data store (but could be wrong if
>>>>>> there's an issue going on here with these
>>>>>> layers). Something like the following
>>>>>> should work:
>>>>>>
>>>>>> > hpcc_data_requests.py -a data -u <name>
>>>>>> -p <pass> -w nmfs_st -f
>>>>>> /path/to/shapefile.zip --type shape
>>>>>> --update overwrite (the shapefile would
>>>>>> need to be named to match the layer you
>>>>>> want to update).
>>>>>>
>>>>>>
>>>>>> Also, this is something that is a little
>>>>>> whacky with the GeoServer REST API (and
>>>>>> should be better documented somewhere
>>>>>> admittedly), but you may need to do a
>>>>>> delete command for both the Layer and the
>>>>>> FeatureType to fully delete it. When you
>>>>>> create a new layer with the Web UI, it
>>>>>> actually makes both the Layer and
>>>>>> FeatureType behind the scenes. Not sure
>>>>>> if this will fix this particular issue,
>>>>>> but commands to do that would be
>>>>>> something like:
>>>>>>
>>>>>> > hpcc_data_requests.py -a delete_layer
>>>>>> -u <name> -p <pass> -w nmfs_st
>>>>>> --layername <layername>
>>>>>> > hpcc_data_requests.py -a
>>>>>> delete_featuretype -u <name> -p <pass> -w
>>>>>> nmfs_st --layername <layername>
>>>>>>
>>>>>> This should fully delete both a layer and
>>>>>> its corresponding featuretype. It's
>>>>>> possible if you did a delete_layer
>>>>>> command with via the script without doing
>>>>>> delete_featuretype as well, some
>>>>>> featuretypes got left behind. Again, an
>>>>>> issue that should really be better
>>>>>> documented in GeoServer docs but
>>>>>> unfortunately I haven't seen it described
>>>>>> anywhere, I happened to find out myself
>>>>>> after having some similar issues.
>>>>>>
>>>>>> I'm looking around the workspace now, and
>>>>>> I think I see some cases where there are
>>>>>> FeatureTypes that aren't associated with
>>>>>> any layers, in particular
>>>>>> 'communities_pgs' and 'communities_pts'
>>>>>> are both FeatureTypes without
>>>>>> corresponding layers. This could be
>>>>>> preventing the data upload from happening
>>>>>> for these layers. I'm just looking at
>>>>>> these two lists:
>>>>>>
>>>>>> https://services.ogc.noaa.gov/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.data.layer.LayerPage
>>>>>> (filtered by 'nmfs_st')
>>>>>> https://services.ogc.noaa.gov/geoserver/rest/workspaces/nmfs_st/datastores/postgis.html
>>>>>>
>>>>>> Also, you should probably do the
>>>>>> delete_featuretype command for
>>>>>> 'communities_pgs20130506' and
>>>>>> 'communities_pts20130506', since they
>>>>>> don't have corresponding Layers. I think
>>>>>> that will get those two in sync.
>>>>>>
>>>>>> I'm not sure why you get such a long list
>>>>>> in the New Layer dialog though, looks
>>>>>> like a bug to me. No idea where
>>>>>> 'SI_SE_Gulf_sites_pg_ALLsites' and
>>>>>> 'SI_SE_Gulf_sites_pt' are coming from,
>>>>>> because those aren't listed as current
>>>>>> FeatureTypes in the REST API.
>>>>>>
>>>>>> HTH,
>>>>>> Micah
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 9/24/2013 9:30 AM, John Kennedy - NOAA
>>>>>> Affiliate wrote:
>>>>>>> The two layers of concern are
>>>>>>> communities_pgs and communities_pts.
>>>>>>> When I attempt to load new data, via a
>>>>>>> zip file with a shape file, I get a
>>>>>>> message that the communities_pgs exists.
>>>>>>> When I look at the list of source under
>>>>>>> the "Add sources" button I can see
>>>>>>> the communities_pgs layer and the
>>>>>>> publish link. But when I click the link
>>>>>>> and attempt to save I get the message
>>>>>>> below. Deleting layer via the hpcc_data
>>>>>>> python script also seems to fail.
>>>>>>>
>>>>>>> Inline image 1
>>>>>>>
>>>>>>> John F. Kennedy, PhD, CPG
>>>>>>> GIS Specialist
>>>>>>> Contractor: Caelum Corp.
>>>>>>> NOAA Fisheries Office of Science and
>>>>>>> Technology
>>>>>>> 1315 East-West Highway
>>>>>>> SSMC3 Rm 9518
>>>>>>> Silver Spring, MD 20910
>>>>>>> 301-427-8149 <tel:301-427-8149>
>>>>>>>
>>>>>>> Comments? Have I been of assistance?
>>>>>>> Please let my supervisors know. :~)
>>>>>>> Tim Haverland - NOAA Federal
>>>>>>> <Tim.Haverland at noaa.gov
>>>>>>> <mailto:Tim.Haverland at noaa.gov>>
>>>>>>> Hussain Jabalpurwala - NOAA Affiliate
>>>>>>> <hussain.jabalpurwala at noaa.gov
>>>>>>> <mailto:hussain.jabalpurwala at noaa.gov>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Sep 24, 2013 at 9:20 AM, Tim
>>>>>>> Haverland - NOAA Federal
>>>>>>> <tim.haverland at noaa.gov
>>>>>>> <mailto:tim.haverland at noaa.gov>> wrote:
>>>>>>>
>>>>>>> I'm adding John Kennedy to this
>>>>>>> thread so he can work with you on
>>>>>>> this issue.
>>>>>>>
>>>>>>> Tim
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Sep 24, 2013 at 8:49 AM,
>>>>>>> Micah Wengren
>>>>>>> <micah.wengren at noaa.gov
>>>>>>> <mailto:micah.wengren at noaa.gov>> wrote:
>>>>>>>
>>>>>>> Tim,
>>>>>>>
>>>>>>> I can see some layers in your
>>>>>>> workspace in the New Layer
>>>>>>> dialog that aren't currently
>>>>>>> published and have a 'Publish'
>>>>>>> link next to them, so I see what
>>>>>>> you mean. Are you able to
>>>>>>> create a new SQL view layer with
>>>>>>> the same name as one of those
>>>>>>> layers, or does that not work?
>>>>>>> Can you give me the name of one
>>>>>>> of the layers that you've tried
>>>>>>> with and I'll pass it on to
>>>>>>> OpenGeo support to troubleshoot,
>>>>>>> that way I can give them a
>>>>>>> specific layer name with the issue.
>>>>>>>
>>>>>>> The layers should be
>>>>>>> disappearing but for whatever
>>>>>>> reason they're not.
>>>>>>>
>>>>>>> Micah
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 9/23/2013 3:55 PM, Tim
>>>>>>> Haverland - NOAA Federal wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> We've been trying to clean up a
>>>>>>>> bunch of layers in our
>>>>>>>> workspace. We can delete layers
>>>>>>>> using the "Remove selected
>>>>>>>> resources" button on the Layers
>>>>>>>> page, but when we try to
>>>>>>>> replace that layer with
>>>>>>>> another, the layer is still there.
>>>>>>>>
>>>>>>>> We can verify this by clicking
>>>>>>>> "Add a new resource" ... all of
>>>>>>>> the old layers are still there
>>>>>>>> and I can click Publish to make
>>>>>>>> them "live" again.
>>>>>>>>
>>>>>>>> How do we permanently get rid
>>>>>>>> of unwanted layers?
>>>>>>>>
>>>>>>>> Tim
>>>>>>>> --
>>>>>>>> *Tim Haverland*
>>>>>>>> Acting Operations Branch Chief
>>>>>>>> NOAA Fisheries Office of
>>>>>>>> Science and Technology
>>>>>>>> 1315 East-West Highway
>>>>>>>> SSMC3 Rm 12303
>>>>>>>> Silver Spring, MD 20910
>>>>>>>> 301-427-8137 <tel:301-427-8137>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Open.ogc mailing list
>>>>>>>> Open.ogc at list.woc.noaa.gov <mailto:Open.ogc at list.woc.noaa.gov>
>>>>>>>> https://list.woc.noaa.gov/cgi-bin/mailman/listinfo/open.ogc
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Tim Haverland*
>>>>>>> Acting Operations Branch Chief
>>>>>>> NOAA Fisheries Office of Science and
>>>>>>> Technology
>>>>>>> 1315 East-West Highway
>>>>>>> SSMC3 Rm 12303
>>>>>>> Silver Spring, MD 20910
>>>>>>> 301-427-8137 <tel:301-427-8137>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130925/5f441a5b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166250 bytes
Desc: not available
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130925/5f441a5b/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 153411 bytes
Desc: not available
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130925/5f441a5b/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 164721 bytes
Desc: not available
URL: <https://list.woc.noaa.gov/pipermail/open.ogc/attachments/20130925/5f441a5b/attachment-0005.png>
More information about the Open.ogc
mailing list